[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:10 dist-test-slave-x4qh krb5kdc[22656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:10 dist-test-slave-x4qh krb5kdc[22656](info): set up 2 sockets
Aug 08 22:50:10 dist-test-slave-x4qh krb5kdc[22656](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 22:50:11.757714 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:11 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:11.863958 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:35811
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:11.949174 22675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:11.949391 22675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:11.949414 22675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:11.950464 22675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:11.950498 22675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:11.950511 22675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:11.950524 22675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:11.950536 22675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:11.952066 22675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:35811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:11.952412 22675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:11.952636 22675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:11.959751 22687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:11.959966 22683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:11.960202 22685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:11.960306 22675 server_base.cc:1034] running on GCE node
I20260808 22:50:11.960614 22675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:11.960989 22675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:11.962157 22675 hybrid_clock.cc:648] HybridClock initialized: now 1786229411962144 us; error 64 us; skew 500 ppm
Aug 08 22:50:11 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229411, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:11.965010 22675 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:11.965656 22675 webserver.cc:460] Webserver started at http://127.22.30.62:42393/ using document root <none> and password file <none>
I20260808 22:50:11.965905 22675 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:11.965950 22675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:11.966084 22675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:11.967178 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "19402a22d1354d10b1c3415e9d66c61e"
format_stamp: "Formatted at 2026-08-08 22:50:11 on dist-test-slave-x4qh"
server_key: "f17302f065f227eb59295e1d083160e4"
server_key_iv: "f1ee496021e201fa5d63b7e4736197fa"
server_key_version: "encryptionkey@0"
I20260808 22:50:11.967549 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "19402a22d1354d10b1c3415e9d66c61e"
format_stamp: "Formatted at 2026-08-08 22:50:11 on dist-test-slave-x4qh"
server_key: "f17302f065f227eb59295e1d083160e4"
server_key_iv: "f1ee496021e201fa5d63b7e4736197fa"
server_key_version: "encryptionkey@0"
I20260808 22:50:11.969115 22675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:11.970053 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:11.970368 22675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:11.970463 22675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "19402a22d1354d10b1c3415e9d66c61e"
format_stamp: "Formatted at 2026-08-08 22:50:11 on dist-test-slave-x4qh"
server_key: "f17302f065f227eb59295e1d083160e4"
server_key_iv: "f1ee496021e201fa5d63b7e4736197fa"
server_key_version: "encryptionkey@0"
I20260808 22:50:11.970534 22675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:11.995931 22675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:11.997871 22675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:11.998056 22675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:12.002717 22675 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:35811
I20260808 22:50:12.002833 22768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:35811 every 8 connection(s)
I20260808 22:50:12.003278 22675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:12.004611 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:12.007586 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e: Bootstrap starting.
I20260808 22:50:12.008549 22769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:12.008915 22769 log.cc:824] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:12.009734 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e: No bootstrap required, opened a new log
I20260808 22:50:12.010175 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22675
I20260808 22:50:12.010259 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:12.010856 22648 external_mini_cluster.cc:1428] Setting key db5928da4fd80dc173037437221b4ace
I20260808 22:50:12.011543 22769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19402a22d1354d10b1c3415e9d66c61e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35811 } }
I20260808 22:50:12.011799 22769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:12.011909 22769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19402a22d1354d10b1c3415e9d66c61e, State: Initialized, Role: FOLLOWER
I20260808 22:50:12.012094 22769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19402a22d1354d10b1c3415e9d66c61e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35811 } }
I20260808 22:50:12.012298 22769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:12.012356 22769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:12.012475 22769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:12.013461 22769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19402a22d1354d10b1c3415e9d66c61e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35811 } }
I20260808 22:50:12.013664 22769 leader_election.cc:302] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19402a22d1354d10b1c3415e9d66c61e; no voters:
I20260808 22:50:12.013931 22769 leader_election.cc:288] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:12.013952 22774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:12.014145 22774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [term 1 LEADER]: Becoming Leader. State: Replica: 19402a22d1354d10b1c3415e9d66c61e, State: Running, Role: LEADER
I20260808 22:50:12.014424 22769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:12.014395 22774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19402a22d1354d10b1c3415e9d66c61e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35811 } }
I20260808 22:50:12.014887 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19402a22d1354d10b1c3415e9d66c61e. Latest consensus state: current_term: 1 leader_uuid: "19402a22d1354d10b1c3415e9d66c61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19402a22d1354d10b1c3415e9d66c61e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35811 } } }
I20260808 22:50:12.014966 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:12.015445 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19402a22d1354d10b1c3415e9d66c61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19402a22d1354d10b1c3415e9d66c61e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35811 } } }
I20260808 22:50:12.015990 22781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:12.016139 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:12.017093 22781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:12.020670 22781 catalog_manager.cc:1347] Generated new cluster ID: 9709c017e1a4465ea489f2d8a9c374b5
I20260808 22:50:12.020744 22781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:12.028028 22781 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:12.028615 22781 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:12.030074 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.012103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.22.30.62:35811)
0808 22:50:12.012375 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:12.012385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:12.012683 (+ 298us) server_negotiation.cc:407] Connection header received
0808 22:50:12.014868 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:12.014875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:12.014975 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:12.015139 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:12.017748 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.019539 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.020642 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.021365 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.021489 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.025649 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:12.025653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:12.025653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:12.025663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:12.027884 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.028385 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.028388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.028388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.028495 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.028827 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.028828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.028828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.029069 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:12.029229 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:12.029524 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 22:50:12.029698 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260808 22:50:12.034078 22781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19402a22d1354d10b1c3415e9d66c61e: Generated new TSK 0
I20260808 22:50:12.034322 22781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:12.063794 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35811
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:12.179385 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:12.179562 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:12.179585 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:12.180725 22805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:12.180765 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:12.180790 22805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:12.182298 22805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:12.182582 22805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:12.182826 22805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:12.190358 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:12.190466 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:12.190626 22805 server_base.cc:1034] running on GCE node
W20260808 22:50:12.190897 22813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:12.191114 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:12.191412 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:12.192564 22805 hybrid_clock.cc:648] HybridClock initialized: now 1786229412192544 us; error 38 us; skew 500 ppm
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:12.195218 22805 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:12.195705 22805 webserver.cc:460] Webserver started at http://127.22.30.1:45569/ using document root <none> and password file <none>
I20260808 22:50:12.195952 22805 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:12.196023 22805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:12.196187 22805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:12.197137 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "00722d76730b47428501476dd2708479"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "eb3018bd74b69ecc9cd3b01a5810b55f"
server_key_iv: "dc65defcf3a7705ced8d05291eb5dfc9"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.197470 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "00722d76730b47428501476dd2708479"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "eb3018bd74b69ecc9cd3b01a5810b55f"
server_key_iv: "dc65defcf3a7705ced8d05291eb5dfc9"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.198689 22805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:12.199437 22828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.199617 22805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:12.199723 22805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "00722d76730b47428501476dd2708479"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "eb3018bd74b69ecc9cd3b01a5810b55f"
server_key_iv: "dc65defcf3a7705ced8d05291eb5dfc9"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.199806 22805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:12.234807 22805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:12.237052 22805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:12.237205 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:12.237462 22805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:12.237890 22805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:12.237987 22805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.238029 22805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:12.238059 22805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.245201 22805 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:34203
I20260808 22:50:12.245323 22978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:34203 every 8 connection(s)
I20260808 22:50:12.245595 22805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:12.249280 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22805
I20260808 22:50:12.249420 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:12.249570 22648 external_mini_cluster.cc:1428] Setting key c11a32975e9cb4e6b6f99a30723a9f75
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:12.256652 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.246474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:38339 (local address 127.22.30.62:35811)
0808 22:50:12.246541 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:12.246543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:12.246789 (+ 246us) server_negotiation.cc:407] Connection header received
0808 22:50:12.247235 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:12.247236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:12.247244 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:12.247298 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:12.248639 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.249166 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.250758 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.250945 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.251000 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.253815 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:12.253817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:12.253817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:12.253824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:12.255125 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.255642 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.255644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.255644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.255706 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.255982 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.255983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.255983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.256097 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:12.256188 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:12.256472 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 22:50:12.256571 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:50:12.256855 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.246420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:35811 (local address 127.22.30.1:38339)
0808 22:50:12.246644 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:12.246700 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:12.247159 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:12.247360 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:12.247363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:12.248250 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:12.248536 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:12.248541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.249262 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.249263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:12.250670 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:12.250674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.251029 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.251030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:12.251228 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.251723 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:12.251733 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:12.253721 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:12.255231 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:12.255237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:12.255239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:12.255554 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:12.255775 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:12.255776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:12.255776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:12.255889 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:12.256182 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:12.256190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:12.256325 (+ 135us) client_negotiation.cc:769] Sending connection context
0808 22:50:12.256436 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 22:50:12.256591 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":140,"threads_started":1}
I20260808 22:50:12.257400 22980 heartbeater.cc:347] Connected to a master server at 127.22.30.62:35811
I20260808 22:50:12.257499 22980 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:12.257688 22980 heartbeater.cc:511] Master 127.22.30.62:35811 requested a full tablet report, sending...
I20260808 22:50:12.258466 22721 ts_manager.cc:194] Registered new tserver with Master: 00722d76730b47428501476dd2708479 (127.22.30.1:34203)
I20260808 22:50:12.259168 22721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:38339
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:12.275777 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35811
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:12.375017 22987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:12.375205 22987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:12.375231 22987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:12.376416 22987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:12.376461 22987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:12.376525 22987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:12.377929 22987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:12.378166 22987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:12.378378 22987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:12.383447 22998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:12.383397 22995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:12.383526 22987 server_base.cc:1034] running on GCE node
W20260808 22:50:12.383390 22994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:12.383800 22987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:12.384037 22987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:12.385202 22987 hybrid_clock.cc:648] HybridClock initialized: now 1786229412385197 us; error 36 us; skew 500 ppm
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:12.387643 22987 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:12.388033 22987 webserver.cc:460] Webserver started at http://127.22.30.2:41011/ using document root <none> and password file <none>
I20260808 22:50:12.388250 22987 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:12.388314 22987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:12.388458 22987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:12.389359 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "125bb9e236b74f14b31c27b9b5334ba3"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "caf0e8806f2b85f2f3170a5d8b031b4e"
server_key_iv: "c317381debd4a417f909f1784e1f771b"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.389698 22987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "125bb9e236b74f14b31c27b9b5334ba3"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "caf0e8806f2b85f2f3170a5d8b031b4e"
server_key_iv: "c317381debd4a417f909f1784e1f771b"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.390895 22987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 22:50:12.391664 23008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.391847 22987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:12.391943 22987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "125bb9e236b74f14b31c27b9b5334ba3"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "caf0e8806f2b85f2f3170a5d8b031b4e"
server_key_iv: "c317381debd4a417f909f1784e1f771b"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.392019 22987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:12.404210 22987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:12.406394 22987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:12.406587 22987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:12.406867 22987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:12.407166 22987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:12.407220 22987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.407259 22987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:12.407282 22987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.414361 22987 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:33273
I20260808 22:50:12.414395 23167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:33273 every 8 connection(s)
I20260808 22:50:12.414736 22987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:12.420264 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22987
I20260808 22:50:12.420342 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:12.420513 22648 external_mini_cluster.cc:1428] Setting key e0dac2aa4501afd8d93d2077a1293164
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:12.424671 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.415494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:43309 (local address 127.22.30.62:35811)
0808 22:50:12.415534 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:12.415535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:12.415977 (+ 442us) server_negotiation.cc:407] Connection header received
0808 22:50:12.416717 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:12.416719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:12.416731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:12.416780 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:12.417776 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.418256 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.419175 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.419420 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.419496 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.421498 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:12.421499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:12.421499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:12.421506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:12.423167 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.423722 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.423723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.423723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.423784 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.424065 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.424066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.424066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.424164 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:12.424214 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:12.424550 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 22:50:12.424611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:12.425057 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.415495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:35811 (local address 127.22.30.2:43309)
0808 22:50:12.415822 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:12.415864 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:12.416604 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:12.416840 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:12.416844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:12.417418 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:12.417667 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:12.417673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.418346 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.418347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:12.419070 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:12.419072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.419507 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.419508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:12.419624 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.419973 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:12.419980 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:12.421403 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:12.423257 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:12.423264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:12.423266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:12.423627 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:12.423856 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:12.423857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:12.423857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:12.423999 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:12.424232 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:12.424238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:12.424416 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 22:50:12.424531 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 22:50:12.424709 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":147,"threads_started":1}
I20260808 22:50:12.425590 23168 heartbeater.cc:347] Connected to a master server at 127.22.30.62:35811
I20260808 22:50:12.425702 23168 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:12.425886 23168 heartbeater.cc:511] Master 127.22.30.62:35811 requested a full tablet report, sending...
I20260808 22:50:12.426259 22721 ts_manager.cc:194] Registered new tserver with Master: 125bb9e236b74f14b31c27b9b5334ba3 (127.22.30.2:33273)
I20260808 22:50:12.426667 22721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:43309
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:12.443498 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35811
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:12.536962 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:12.537151 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:12.537189 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:12.538298 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:12.538360 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:12.538419 23174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:12.539798 23174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35811
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:12.540021 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:12.540220 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:12.545418 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:12.545460 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:12.545468 23174 server_base.cc:1034] running on GCE node
W20260808 22:50:12.545431 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:12.545815 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:12.546058 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:12.547230 23174 hybrid_clock.cc:648] HybridClock initialized: now 1786229412547217 us; error 42 us; skew 500 ppm
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:12.549726 23174 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:12.550161 23174 webserver.cc:460] Webserver started at http://127.22.30.3:41227/ using document root <none> and password file <none>
I20260808 22:50:12.550333 23174 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:12.550369 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:12.550447 23174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:12.551327 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "7b685ac85cf242d09647a1d7845a77a2"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "8610fea0267cdfebdb728b6b7a18e187"
server_key_iv: "e53166252b2b4f805d6e91cb0414f638"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.551681 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "7b685ac85cf242d09647a1d7845a77a2"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "8610fea0267cdfebdb728b6b7a18e187"
server_key_iv: "e53166252b2b4f805d6e91cb0414f638"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.552791 23174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:12.553459 23195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.553604 23174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.553682 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "7b685ac85cf242d09647a1d7845a77a2"
format_stamp: "Formatted at 2026-08-08 22:50:12 on dist-test-slave-x4qh"
server_key: "8610fea0267cdfebdb728b6b7a18e187"
server_key_iv: "e53166252b2b4f805d6e91cb0414f638"
server_key_version: "encryptionkey@0"
I20260808 22:50:12.553733 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:12.563730 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:12.565569 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:12.565678 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:12.565923 23174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:12.566265 23174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:12.566325 23174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.566344 23174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:12.566444 23174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:12.576869 23174 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:37259
I20260808 22:50:12.577016 23368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:37259 every 8 connection(s)
I20260808 22:50:12.577312 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:12.578352 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23174
I20260808 22:50:12.578433 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:12.578603 22648 external_mini_cluster.cc:1428] Setting key ac3ad48a0c56f5c1f158a1415032cbad
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:12.586076 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.578064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:33733 (local address 127.22.30.62:35811)
0808 22:50:12.578121 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:12.578123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:12.578382 (+ 259us) server_negotiation.cc:407] Connection header received
0808 22:50:12.579071 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:12.579072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:12.579083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:12.579139 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:12.580205 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.580702 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.581637 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.581841 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.581888 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.583788 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:12.583789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:12.583790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:12.583795 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:12.585008 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.585365 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.585366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.585366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.585412 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.585611 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.585611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.585612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.585702 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:12.585749 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:12.585963 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 22:50:12.586018 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:50:12.586210 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.578037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:35811 (local address 127.22.30.3:33733)
0808 22:50:12.578227 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:12.578273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:12.578971 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:12.579204 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:12.579208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:12.579929 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:12.580109 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:12.580114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.580786 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.580787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:12.581557 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:12.581559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.581910 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.581910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:12.582017 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.582411 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:12.582418 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:12.583704 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:12.585083 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:12.585089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:12.585091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:12.585294 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:12.585470 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:12.585471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:12.585471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:12.585558 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:12.585754 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:12.585758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:12.585861 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 22:50:12.585935 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:50:12.586039 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260808 22:50:12.586673 23369 heartbeater.cc:347] Connected to a master server at 127.22.30.62:35811
I20260808 22:50:12.586773 23369 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:12.586922 23369 heartbeater.cc:511] Master 127.22.30.62:35811 requested a full tablet report, sending...
I20260808 22:50:12.587273 22721 ts_manager.cc:194] Registered new tserver with Master: 7b685ac85cf242d09647a1d7845a77a2 (127.22.30.3:37259)
I20260808 22:50:12.587622 22721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:33733
I20260808 22:50:12.590490 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:12.613288 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.606314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.22.30.1:34203)
0808 22:50:12.606563 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:12.606567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:12.606576 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:12.606600 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:12.606602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:12.606614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:12.606674 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:12.607154 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.607857 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.608519 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.608789 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.608857 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.610956 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:12.610964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:12.610964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:12.610970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:12.612326 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.612608 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.612610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.612610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.612657 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.612818 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.612819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.612819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.612942 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:12.613060 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:12.613107 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 22:50:12.613163 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":165,"threads_started":1}
W20260808 22:50:12.613715 22863 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44820
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[22656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:12.635392 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:12.628661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.22.30.1:34203)
0808 22:50:12.628731 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:12.628733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:12.628837 (+ 104us) server_negotiation.cc:407] Connection header received
0808 22:50:12.628875 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:12.628876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:12.628887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:12.628938 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:12.629475 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.629966 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.630658 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:12.630852 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:12.630911 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:12.633016 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:12.633017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:12.633018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:12.633027 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:12.634189 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.634546 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.634547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.634548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.634622 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:12.634857 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:12.634858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:12.634858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:12.634960 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:12.635085 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:12.635189 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 22:50:12.635267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:50:12.635921 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22805
I20260808 22:50:12.650992 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22987
I20260808 22:50:12.664587 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23174
I20260808 22:50:12.673827 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22675
2026-08-08T22:50:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2099 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[23388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[23388](info): set up 2 sockets
Aug 08 22:50:12 dist-test-slave-x4qh krb5kdc[23388](info): commencing operation
krb5kdc: starting...
W20260808 22:50:13.837428 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:13 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:13.943959 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37313
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:14.021945 23408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:14.022145 23408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:14.022166 23408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:14.023682 23408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:14.023749 23408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:14.023777 23408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:14.023802 23408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:14.023836 23408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:14.025312 23408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:14.025494 23408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:14.025671 23408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:14.030588 23408 server_base.cc:1034] running on GCE node
W20260808 22:50:14.030537 23419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.030524 23417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.030517 23415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:14.030928 23408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:14.031224 23408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:14.032392 23408 hybrid_clock.cc:648] HybridClock initialized: now 1786229414032384 us; error 39 us; skew 500 ppm
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:14.038671 23408 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:14.039029 23408 webserver.cc:460] Webserver started at http://127.22.30.62:44677/ using document root <none> and password file <none>
I20260808 22:50:14.039237 23408 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:14.039307 23408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:14.039477 23408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:14.040520 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "0652c0ec8b604462ada28be4cd02ce01"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "c4c8e7de2adddba4ac43109f62373967"
server_key_iv: "5ab2eed113c82b426ac710cf7698c111"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.040963 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "0652c0ec8b604462ada28be4cd02ce01"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "c4c8e7de2adddba4ac43109f62373967"
server_key_iv: "5ab2eed113c82b426ac710cf7698c111"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.042332 23408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:14.043063 23429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.043255 23408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:14.043365 23408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "0652c0ec8b604462ada28be4cd02ce01"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "c4c8e7de2adddba4ac43109f62373967"
server_key_iv: "5ab2eed113c82b426ac710cf7698c111"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.043452 23408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:14.061563 23408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:14.063199 23408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:14.063354 23408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:14.067091 23408 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:37313
I20260808 22:50:14.067170 23515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:37313 every 8 connection(s)
I20260808 22:50:14.067526 23408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:14.068038 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23408
I20260808 22:50:14.068138 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:14.068312 22648 external_mini_cluster.cc:1428] Setting key eee2cdf400f7f18e86693ab5481d134d
I20260808 22:50:14.068500 23517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:14.071754 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01: Bootstrap starting.
I20260808 22:50:14.072346 23517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:14.072669 23517 log.cc:824] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:14.073482 23517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01: No bootstrap required, opened a new log
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:14.075739 23517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652c0ec8b604462ada28be4cd02ce01" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37313 } }
I20260808 22:50:14.075929 23517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:14.076025 23517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0652c0ec8b604462ada28be4cd02ce01, State: Initialized, Role: FOLLOWER
I20260808 22:50:14.076149 23517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652c0ec8b604462ada28be4cd02ce01" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37313 } }
I20260808 22:50:14.076232 23517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:14.076263 23517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:14.076330 23517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:14.076967 23517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652c0ec8b604462ada28be4cd02ce01" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37313 } }
I20260808 22:50:14.077075 23517 leader_election.cc:302] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0652c0ec8b604462ada28be4cd02ce01; no voters:
I20260808 22:50:14.077308 23517 leader_election.cc:288] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:14.077345 23524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:14.077457 23524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [term 1 LEADER]: Becoming Leader. State: Replica: 0652c0ec8b604462ada28be4cd02ce01, State: Running, Role: LEADER
I20260808 22:50:14.077553 23524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652c0ec8b604462ada28be4cd02ce01" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37313 } }
I20260808 22:50:14.077585 23517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:14.077837 23524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0652c0ec8b604462ada28be4cd02ce01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652c0ec8b604462ada28be4cd02ce01" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37313 } } }
I20260808 22:50:14.077907 23524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:14.078022 23524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0652c0ec8b604462ada28be4cd02ce01. Latest consensus state: current_term: 1 leader_uuid: "0652c0ec8b604462ada28be4cd02ce01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0652c0ec8b604462ada28be4cd02ce01" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37313 } } }
I20260808 22:50:14.078088 23524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01 [sys.catalog]: This master's current role is: LEADER
W20260808 22:50:14.079322 23546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:50:14.079401 23546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:50:14.079511 23530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:14.079650 23530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:14.079722 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.068831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.22.30.62:37313)
0808 22:50:14.069042 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.069051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.069084 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:50:14.069633 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.069642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.069698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.069840 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.070449 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.071812 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.072627 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.073176 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.073294 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.075912 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.075916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.075916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.075942 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.077927 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.078352 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.078354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.078355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.078475 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.078713 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.078714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.078714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.079063 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.079258 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.079328 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.079488 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20260808 22:50:14.081166 23530 catalog_manager.cc:1347] Generated new cluster ID: a5169defa318490cbe32d0ba8d5ea22b
I20260808 22:50:14.081216 23530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:14.095443 23530 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:14.095959 23530 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:14.101195 23530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0652c0ec8b604462ada28be4cd02ce01: Generated new TSK 0
I20260808 22:50:14.101337 23530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:14.127804 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37313
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:14.203325 23556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:14.203590 23556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:14.203629 23556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:14.205106 23556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:14.205168 23556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:14.205214 23556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:14.206660 23556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37313
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:14.206924 23556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:14.207137 23556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:14.211784 23565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.211776 23569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:14.211822 23556 server_base.cc:1034] running on GCE node
W20260808 22:50:14.211760 23563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:14.212173 23556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:14.212414 23556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:14.213570 23556 hybrid_clock.cc:648] HybridClock initialized: now 1786229414213558 us; error 46 us; skew 500 ppm
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:14.215895 23556 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:14.216342 23556 webserver.cc:460] Webserver started at http://127.22.30.1:45173/ using document root <none> and password file <none>
I20260808 22:50:14.216578 23556 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:14.216655 23556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:14.216784 23556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:14.217660 23556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "01b5493c35bb4fecaecbca71a78b3481"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "d1d862875e26d046e7a493f204532e25"
server_key_iv: "88cced9fc7dac297b26c06a324b3fc68"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.218000 23556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "01b5493c35bb4fecaecbca71a78b3481"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "d1d862875e26d046e7a493f204532e25"
server_key_iv: "88cced9fc7dac297b26c06a324b3fc68"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.219228 23556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:14.219982 23579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.220166 23556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.220255 23556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "01b5493c35bb4fecaecbca71a78b3481"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "d1d862875e26d046e7a493f204532e25"
server_key_iv: "88cced9fc7dac297b26c06a324b3fc68"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.220336 23556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:14.229781 23556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:14.231320 23556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:14.231477 23556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:14.231703 23556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:14.231977 23556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:14.232026 23556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.232064 23556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:14.232087 23556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.237604 23556 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:40689
I20260808 22:50:14.237638 23742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:40689 every 8 connection(s)
I20260808 22:50:14.237936 23556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:14.241914 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23556
I20260808 22:50:14.242012 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:14.242162 22648 external_mini_cluster.cc:1428] Setting key fbf248ad740cfa6ccd8eb9d82e79040f
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:14.247777 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.238610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:58921 (local address 127.22.30.62:37313)
0808 22:50:14.238653 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.238655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.238843 (+ 188us) server_negotiation.cc:407] Connection header received
0808 22:50:14.239843 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.239844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.239853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.239900 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.240376 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.240846 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.241844 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.242169 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.242244 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.244813 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.244815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.244815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.244823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.246289 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.246767 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.246768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.246768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.246831 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.247103 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.247104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.247105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.247222 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.247311 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.247605 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.247689 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:14.247969 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.238594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37313 (local address 127.22.30.1:58921)
0808 22:50:14.238731 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.238760 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.239749 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.239945 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.239947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.240156 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:14.240313 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.240317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.240947 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.240947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.241756 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.241758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.242263 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.242265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.242409 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.242989 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:14.242996 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:14.244710 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:14.246384 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:14.246391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:14.246393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:14.246653 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:14.246899 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:14.246900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:14.246900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:14.247049 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:14.247306 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:14.247313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:14.247477 (+ 164us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.247584 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.247730 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260808 22:50:14.248479 23743 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37313
I20260808 22:50:14.248600 23743 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:14.248824 23743 heartbeater.cc:511] Master 127.22.30.62:37313 requested a full tablet report, sending...
I20260808 22:50:14.249346 23455 ts_manager.cc:194] Registered new tserver with Master: 01b5493c35bb4fecaecbca71a78b3481 (127.22.30.1:40689)
I20260808 22:50:14.249935 23455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:58921
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:14.265849 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37313
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:14.343204 23749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:14.343396 23749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:14.343418 23749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:14.344424 23749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:14.344457 23749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:14.344482 23749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:14.345975 23749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37313
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:14.346148 23749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:14.346381 23749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:14.351152 23757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.351166 23760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.351166 23758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:14.351395 23749 server_base.cc:1034] running on GCE node
I20260808 22:50:14.351615 23749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:14.351794 23749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:14.352927 23749 hybrid_clock.cc:648] HybridClock initialized: now 1786229414352911 us; error 32 us; skew 500 ppm
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:14.355460 23749 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:14.355906 23749 webserver.cc:460] Webserver started at http://127.22.30.2:35037/ using document root <none> and password file <none>
I20260808 22:50:14.356155 23749 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:14.356225 23749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:14.356376 23749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:14.357347 23749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "9cf69cc201b3420da9ff27162e73cf8e"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "4057673e337e6733e455c3a83d4107b4"
server_key_iv: "8e8e32b55b7f4d7e6f5fbe25e6f7015f"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.357688 23749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "9cf69cc201b3420da9ff27162e73cf8e"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "4057673e337e6733e455c3a83d4107b4"
server_key_iv: "8e8e32b55b7f4d7e6f5fbe25e6f7015f"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.359004 23749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:14.359784 23771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.359956 23749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.360054 23749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "9cf69cc201b3420da9ff27162e73cf8e"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "4057673e337e6733e455c3a83d4107b4"
server_key_iv: "8e8e32b55b7f4d7e6f5fbe25e6f7015f"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.360134 23749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:14.373350 23749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:14.375641 23749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:14.375795 23749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:14.375999 23749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:14.376289 23749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:14.376341 23749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.376379 23749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:14.376406 23749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.382289 23749 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:37171
I20260808 22:50:14.382318 23933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:37171 every 8 connection(s)
I20260808 22:50:14.382618 23749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:14.390533 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23749
I20260808 22:50:14.390630 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:14.390790 22648 external_mini_cluster.cc:1428] Setting key 6a7d4d1419544d19ce7fe982176b2d9e
I20260808 22:50:14.390883 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.383296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:34919 (local address 127.22.30.62:37313)
0808 22:50:14.383396 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.383399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.383625 (+ 226us) server_negotiation.cc:407] Connection header received
0808 22:50:14.384477 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.384478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.384488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.384539 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.385072 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.385562 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.386487 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.386656 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.386712 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.388531 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.388533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.388533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.388539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.389654 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.389998 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.389999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.389999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.390046 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.390240 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.390241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.390241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.390340 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.390385 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.390730 (+ 345us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.390790 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:50:14.391062 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.383246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37313 (local address 127.22.30.2:34919)
0808 22:50:14.383481 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.383534 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.384390 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.384589 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.384592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.384834 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:14.385003 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.385008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.385638 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.385638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.386417 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.386419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.386733 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.386734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.386827 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.387167 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:14.387174 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:14.388447 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:14.389724 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:14.389730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:14.389731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:14.389937 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:14.390102 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:14.390103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:14.390103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:14.390189 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:14.390400 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:14.390406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:14.390547 (+ 141us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.390688 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.390853 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":144,"threads_started":1}
I20260808 22:50:14.391587 23934 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37313
I20260808 22:50:14.391678 23934 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:14.391834 23934 heartbeater.cc:511] Master 127.22.30.62:37313 requested a full tablet report, sending...
I20260808 22:50:14.392206 23455 ts_manager.cc:194] Registered new tserver with Master: 9cf69cc201b3420da9ff27162e73cf8e (127.22.30.2:37171)
I20260808 22:50:14.392546 23455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:34919
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:14.413189 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37313
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:14.493245 23939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:14.493438 23939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:14.493459 23939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:14.494439 23939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:14.494467 23939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:14.494489 23939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:14.496026 23939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37313
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:14.496199 23939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:14.496379 23939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:14.501405 23951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.501387 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:14.501385 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:14.501718 23939 server_base.cc:1034] running on GCE node
I20260808 22:50:14.501951 23939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:14.502178 23939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:14.503336 23939 hybrid_clock.cc:648] HybridClock initialized: now 1786229414503321 us; error 39 us; skew 500 ppm
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:14.505808 23939 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:14.506249 23939 webserver.cc:460] Webserver started at http://127.22.30.3:35191/ using document root <none> and password file <none>
I20260808 22:50:14.506431 23939 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:14.506470 23939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:14.506557 23939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:14.507586 23939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "02d4f7bae48c4044ad2e691ad81e94fe"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "0f393cc138e90995b841a265fc9d06ff"
server_key_iv: "f085d9f9d609b117f297e6f6be2599e7"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.507891 23939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "02d4f7bae48c4044ad2e691ad81e94fe"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "0f393cc138e90995b841a265fc9d06ff"
server_key_iv: "f085d9f9d609b117f297e6f6be2599e7"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.509104 23939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:14.509840 23965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.510035 23939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:14.510115 23939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "02d4f7bae48c4044ad2e691ad81e94fe"
format_stamp: "Formatted at 2026-08-08 22:50:14 on dist-test-slave-x4qh"
server_key: "0f393cc138e90995b841a265fc9d06ff"
server_key_iv: "f085d9f9d609b117f297e6f6be2599e7"
server_key_version: "encryptionkey@0"
I20260808 22:50:14.510165 23939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:14.526912 23939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:14.529426 23939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:14.529587 23939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:14.529875 23939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:14.530228 23939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:14.530303 23939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.530340 23939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:14.530364 23939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:14.543143 23939 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:43029
I20260808 22:50:14.543229 24127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:43029 every 8 connection(s)
I20260808 22:50:14.543597 23939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:14.548866 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23939
I20260808 22:50:14.548976 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:14.549172 22648 external_mini_cluster.cc:1428] Setting key 251316eb12c323bf926b884fd6b72cd5
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:14.553485 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.544335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:54963 (local address 127.22.30.62:37313)
0808 22:50:14.544375 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.544377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.544600 (+ 223us) server_negotiation.cc:407] Connection header received
0808 22:50:14.545661 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.545663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.545677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.545730 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.546370 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.546902 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.548082 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.548271 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.548357 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.550790 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.550791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.550791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.550799 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.552347 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.552681 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.552682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.552683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.552738 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.552938 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.552939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.552939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.553039 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.553116 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.553354 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.553424 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:14.553646 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.544298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37313 (local address 127.22.30.3:54963)
0808 22:50:14.544480 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.544515 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.545524 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.545805 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.545809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.546064 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:14.546253 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.546258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.547006 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.547007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.547980 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.547983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.548364 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.548366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.548491 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.549077 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:14.549085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:14.550707 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:14.552427 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:14.552432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:14.552434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:14.552621 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:14.552802 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:14.552802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:14.552802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:14.552887 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:14.553124 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:14.553128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:14.553233 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.553323 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.553438 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260808 22:50:14.554100 24128 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37313
I20260808 22:50:14.554198 24128 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:14.554337 24128 heartbeater.cc:511] Master 127.22.30.62:37313 requested a full tablet report, sending...
I20260808 22:50:14.554754 23455 ts_manager.cc:194] Registered new tserver with Master: 02d4f7bae48c4044ad2e691ad81e94fe (127.22.30.3:43029)
I20260808 22:50:14.555080 23455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:54963
I20260808 22:50:14.561022 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:50:14.568331 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.562901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.22.30.62:37313)
0808 22:50:14.562950 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.562951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.563056 (+ 105us) server_negotiation.cc:407] Connection header received
0808 22:50:14.563136 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.563137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.563144 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.563202 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.563856 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.564266 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.564912 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.565102 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.565149 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.566200 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.566201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.566202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.566207 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.567511 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.567803 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.567804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.567804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.567849 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.567981 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.567981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.567981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.568084 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.568144 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.568221 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.568271 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:50:14.570616 23455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:14.572501 23455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:14.579439 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.575532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.22.30.2:37171)
0808 22:50:14.575706 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.575710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.575719 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:14.575827 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.575830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.575846 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.575931 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:14.576369 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.577355 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.578385 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.579037 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.579110 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.579168 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.579309 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.579348 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260808 22:50:14.579452 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.575232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.22.30.1:40689)
0808 22:50:14.575475 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.575480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.575491 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:50:14.575694 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.575698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.575716 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.576021 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:14.576136 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.576848 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.578362 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.579094 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.579155 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.579200 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.579307 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.579348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260808 22:50:14.579454 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.575297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37171 (local address 127.0.0.1:57280)
0808 22:50:14.575651 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.575662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.575731 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.575943 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.575944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.575946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:14.575983 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.575984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.577444 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.577446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.578279 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.578282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.579111 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.579112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.579188 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.579203 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.579276 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.579340 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":49,"threads_started":1}
I20260808 22:50:14.579465 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.575095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:40689 (local address 127.0.0.1:48958)
0808 22:50:14.575244 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.575277 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.575335 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.575838 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.575840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.575844 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:14.575923 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.575925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.577054 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.577055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.578268 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.578272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.579153 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.579154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.579208 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.579211 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.579283 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.579340 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":58,"thread_start_us":59,"threads_started":1}
I20260808 22:50:14.580370 23840 tablet_service.cc:1467] Processing CreateTablet for tablet 3502121ffca14b43b00fba09682ec116 (DEFAULT_TABLE table=test-table [id=fdbb2c36d7fc49b3b4b8a02d56a33adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:14.580643 23651 tablet_service.cc:1467] Processing CreateTablet for tablet 3502121ffca14b43b00fba09682ec116 (DEFAULT_TABLE table=test-table [id=fdbb2c36d7fc49b3b4b8a02d56a33adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:14.580746 23840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3502121ffca14b43b00fba09682ec116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:14.580891 23651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3502121ffca14b43b00fba09682ec116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:14.583671 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.575095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:43029 (local address 127.0.0.1:46030)
0808 22:50:14.575430 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.575439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.575482 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.575999 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.576000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.576003 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:14.576065 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.576067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.577327 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.577328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.578346 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.578349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.583407 (+ 5058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.583408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.583500 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.583504 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.583555 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.583599 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":3,"thread_start_us":54,"threads_started":1}
I20260808 22:50:14.583714 24154 tablet_bootstrap.cc:492] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: Bootstrap starting.
I20260808 22:50:14.583937 24152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.575295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.22.30.3:43029)
0808 22:50:14.575839 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.575843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.575851 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:14.575883 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.575885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.575899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.576142 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:14.576213 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.577223 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.578581 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.580101 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.580234 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.580287 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.583803 (+ 3516us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.583845 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":107,"threads_started":1}
I20260808 22:50:14.584355 24157 tablet_bootstrap.cc:492] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: Bootstrap starting.
I20260808 22:50:14.584425 24154 tablet_bootstrap.cc:654] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:14.584641 24033 tablet_service.cc:1467] Processing CreateTablet for tablet 3502121ffca14b43b00fba09682ec116 (DEFAULT_TABLE table=test-table [id=fdbb2c36d7fc49b3b4b8a02d56a33adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:14.584921 24157 tablet_bootstrap.cc:654] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:14.585078 24033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3502121ffca14b43b00fba09682ec116. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:14.585182 24154 log.cc:824] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:14.585498 24157 log.cc:824] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:14.585974 24154 tablet_bootstrap.cc:492] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: No bootstrap required, opened a new log
I20260808 22:50:14.586061 24154 ts_tablet_manager.cc:1397] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:14.586212 24157 tablet_bootstrap.cc:492] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: No bootstrap required, opened a new log
I20260808 22:50:14.586288 24157 ts_tablet_manager.cc:1397] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:14.587802 24157 raft_consensus.cc:348] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.587953 24157 raft_consensus.cc:374] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:14.587996 24157 raft_consensus.cc:729] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b5493c35bb4fecaecbca71a78b3481, State: Initialized, Role: FOLLOWER
I20260808 22:50:14.588096 24157 consensus_queue.cc:260] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.588402 24157 ts_tablet_manager.cc:1428] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:14.588408 24154 raft_consensus.cc:348] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.588613 24154 raft_consensus.cc:374] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:14.588639 23743 heartbeater.cc:503] Master 127.22.30.62:37313 was elected leader, sending a full tablet report...
I20260808 22:50:14.588672 24154 raft_consensus.cc:729] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf69cc201b3420da9ff27162e73cf8e, State: Initialized, Role: FOLLOWER
I20260808 22:50:14.588567 24161 tablet_bootstrap.cc:492] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: Bootstrap starting.
I20260808 22:50:14.588810 24154 consensus_queue.cc:260] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.589118 23934 heartbeater.cc:503] Master 127.22.30.62:37313 was elected leader, sending a full tablet report...
I20260808 22:50:14.589118 24154 ts_tablet_manager.cc:1428] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:50:14.589428 24161 tablet_bootstrap.cc:654] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:14.589993 24161 log.cc:824] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:14.590605 24161 tablet_bootstrap.cc:492] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: No bootstrap required, opened a new log
I20260808 22:50:14.590653 24161 ts_tablet_manager.cc:1397] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:14.592046 24161 raft_consensus.cc:348] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.592159 24161 raft_consensus.cc:374] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:14.592177 24161 raft_consensus.cc:729] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d4f7bae48c4044ad2e691ad81e94fe, State: Initialized, Role: FOLLOWER
I20260808 22:50:14.592278 24161 consensus_queue.cc:260] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.592521 24161 ts_tablet_manager.cc:1428] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:14.592602 24128 heartbeater.cc:503] Master 127.22.30.62:37313 was elected leader, sending a full tablet report...
W20260808 22:50:14.633122 23935 tablet.cc:2369] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:14.714152 24166 raft_consensus.cc:482] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:14.714289 24166 raft_consensus.cc:504] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.714895 24166 leader_election.cc:288] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01b5493c35bb4fecaecbca71a78b3481 (127.22.30.1:40689), 9cf69cc201b3420da9ff27162e73cf8e (127.22.30.2:37171)
I20260808 22:50:14.717787 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.715106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:40689 (local address 127.22.30.3:58665)
0808 22:50:14.715161 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.715170 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.715239 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.715441 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.715442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.715444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:14.715592 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.715594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.716229 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.716230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.716956 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.716958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.717508 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.717509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.717587 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.717590 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.717672 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.717713 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:50:14.717777 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.715168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:58665 (local address 127.22.30.1:40689)
0808 22:50:14.715236 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.715239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.715248 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:14.715295 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.715296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.715308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.715354 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:14.715673 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.716144 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.717042 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.717422 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.717506 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.717517 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.717678 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.717713 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:50:14.718020 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.715305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37171 (local address 127.22.30.3:51663)
0808 22:50:14.715496 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:14.715504 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:14.715608 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:14.715757 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:14.715758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:14.715759 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:14.715838 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.715840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.716452 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.716453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.717173 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:14.717174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.717714 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.717715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:14.717787 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.717791 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:14.717871 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 22:50:14.717915 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":132,"threads_started":1}
I20260808 22:50:14.718137 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.715399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:51663 (local address 127.22.30.2:37171)
0808 22:50:14.715446 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.715448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.715592 (+ 144us) server_negotiation.cc:407] Connection header received
0808 22:50:14.715672 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.715673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.715692 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.715752 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:14.715920 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.716369 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.717260 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.717632 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.717712 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.717720 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.718034 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.718076 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 22:50:14.718180 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3502121ffca14b43b00fba09682ec116" candidate_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b5493c35bb4fecaecbca71a78b3481" is_pre_election: true
I20260808 22:50:14.718405 23676 raft_consensus.cc:2454] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d4f7bae48c4044ad2e691ad81e94fe in term 0.
I20260808 22:50:14.718405 23866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3502121ffca14b43b00fba09682ec116" candidate_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf69cc201b3420da9ff27162e73cf8e" is_pre_election: true
I20260808 22:50:14.718585 23866 raft_consensus.cc:2454] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d4f7bae48c4044ad2e691ad81e94fe in term 0.
I20260808 22:50:14.718714 23966 leader_election.cc:302] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b5493c35bb4fecaecbca71a78b3481, 02d4f7bae48c4044ad2e691ad81e94fe; no voters:
I20260808 22:50:14.718890 24166 raft_consensus.cc:2781] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:14.718940 24166 raft_consensus.cc:482] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:14.718978 24166 raft_consensus.cc:3037] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:14.719969 24166 raft_consensus.cc:504] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.720095 24166 leader_election.cc:288] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [CANDIDATE]: Term 1 election: Requested vote from peers 01b5493c35bb4fecaecbca71a78b3481 (127.22.30.1:40689), 9cf69cc201b3420da9ff27162e73cf8e (127.22.30.2:37171)
I20260808 22:50:14.720299 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3502121ffca14b43b00fba09682ec116" candidate_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01b5493c35bb4fecaecbca71a78b3481"
I20260808 22:50:14.720307 23866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3502121ffca14b43b00fba09682ec116" candidate_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf69cc201b3420da9ff27162e73cf8e"
I20260808 22:50:14.720373 23866 raft_consensus.cc:3037] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:14.720373 23676 raft_consensus.cc:3037] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:14.721560 23676 raft_consensus.cc:2454] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d4f7bae48c4044ad2e691ad81e94fe in term 1.
I20260808 22:50:14.721560 23866 raft_consensus.cc:2454] T 3502121ffca14b43b00fba09682ec116 P 9cf69cc201b3420da9ff27162e73cf8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d4f7bae48c4044ad2e691ad81e94fe in term 1.
I20260808 22:50:14.721729 23966 leader_election.cc:302] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b5493c35bb4fecaecbca71a78b3481, 02d4f7bae48c4044ad2e691ad81e94fe; no voters:
I20260808 22:50:14.721822 24166 raft_consensus.cc:2781] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:14.721949 24166 raft_consensus.cc:686] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [term 1 LEADER]: Becoming Leader. State: Replica: 02d4f7bae48c4044ad2e691ad81e94fe, State: Running, Role: LEADER
I20260808 22:50:14.722046 24166 consensus_queue.cc:237] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } }
I20260808 22:50:14.722894 23455 catalog_manager.cc:5495] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d4f7bae48c4044ad2e691ad81e94fe (127.22.30.3). New cstate: current_term: 1 leader_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b5493c35bb4fecaecbca71a78b3481" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d4f7bae48c4044ad2e691ad81e94fe" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 43029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf69cc201b3420da9ff27162e73cf8e" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37171 } health_report { overall_health: UNKNOWN } } }
W20260808 22:50:14.738730 23744 tablet.cc:2369] T 3502121ffca14b43b00fba09682ec116 P 01b5493c35bb4fecaecbca71a78b3481: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:14.762290 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.755430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.22.30.1:40689)
0808 22:50:14.755485 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.755487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.755580 (+ 93us) server_negotiation.cc:407] Connection header received
0808 22:50:14.755618 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.755619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.755627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.755676 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.756175 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.756624 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.757233 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.757396 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.757440 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.759719 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.759727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.759727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.759734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.761343 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.761637 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.761645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.761646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.761705 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.761876 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.761877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.761877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.761988 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.762092 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.762173 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.762230 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:14.785161 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.778940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.22.30.1:40689)
0808 22:50:14.778987 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.778989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.779097 (+ 108us) server_negotiation.cc:407] Connection header received
0808 22:50:14.779178 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.779179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.779187 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.779229 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.779765 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.780162 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.780767 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.780941 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.781008 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.782838 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.782840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.782840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.782846 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.783996 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.784352 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.784353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.784353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.784406 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.784603 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.784604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.784604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.784699 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.784794 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.784973 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.785044 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260808 22:50:14.794350 24129 tablet.cc:2369] T 3502121ffca14b43b00fba09682ec116 P 02d4f7bae48c4044ad2e691ad81e94fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[23388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:14.805405 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:14.799502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.22.30.1:40689)
0808 22:50:14.799542 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:14.799544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:14.799660 (+ 116us) server_negotiation.cc:407] Connection header received
0808 22:50:14.799704 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:14.799705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:14.799712 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:14.799758 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:14.800215 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.800656 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.801309 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:14.801468 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:14.801510 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:14.803488 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:14.803488 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:14.803489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:14.803494 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:14.804577 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.804859 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.804860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.804860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.804904 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:14.805046 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:14.805047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:14.805047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:14.805115 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:14.805175 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:14.805280 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 22:50:14.805329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 22:50:14.806041 23621 tablet_service.cc:2908] Rejecting scan request for tablet 3502121ffca14b43b00fba09682ec116: Uninitialized: clean time has not yet been initialized
I20260808 22:50:14.806951 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23556
I20260808 22:50:14.819948 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23749
I20260808 22:50:14.825130 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23939
I20260808 22:50:14.833847 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23408
2026-08-08T22:50:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2159 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[24189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[24189](info): set up 2 sockets
Aug 08 22:50:14 dist-test-slave-x4qh krb5kdc[24189](info): commencing operation
krb5kdc: starting...
W20260808 22:50:15.985734 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:16.090199 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:36203
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:16.166786 24210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:16.167004 24210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:16.167038 24210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:16.168254 24210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:16.168323 24210 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 22:50:16.168345 24210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:16.168366 24210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:16.168398 24210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:16.168421 24210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:16.169836 24210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:36203
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:16.170089 24210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:16.170321 24210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:16.174892 24216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:16.174929 24217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:16.174914 24210 server_base.cc:1034] running on GCE node
W20260808 22:50:16.175019 24219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:16.175355 24210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:16.175741 24210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:16.176936 24210 hybrid_clock.cc:648] HybridClock initialized: now 1786229416176931 us; error 35 us; skew 500 ppm
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:16.179123 24210 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:16.179591 24210 webserver.cc:460] Webserver started at http://127.22.30.62:44769/ using document root <none> and password file <none>
I20260808 22:50:16.179807 24210 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:16.179872 24210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:16.180009 24210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:16.180989 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "05dc2264f81c4d568a654aedb750cc23"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "5097297f6c5f4bb7f1eca98f3f14d206"
server_key_iv: "045b579cc4e705df89aeb82549060628"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.181354 24210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "05dc2264f81c4d568a654aedb750cc23"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "5097297f6c5f4bb7f1eca98f3f14d206"
server_key_iv: "045b579cc4e705df89aeb82549060628"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.182672 24210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:16.183346 24229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.183516 24210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.183609 24210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "05dc2264f81c4d568a654aedb750cc23"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "5097297f6c5f4bb7f1eca98f3f14d206"
server_key_iv: "045b579cc4e705df89aeb82549060628"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.183688 24210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:16.190702 24210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:16.192454 24210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:16.192611 24210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:16.197431 24210 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:36203
I20260808 22:50:16.197490 24319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:36203 every 8 connection(s)
I20260808 22:50:16.197772 24210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:16.198316 24320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.200335 24320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: Bootstrap starting.
I20260808 22:50:16.200906 24320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.201190 24320 log.cc:824] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:16.201838 24320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: No bootstrap required, opened a new log
I20260808 22:50:16.204001 24320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dc2264f81c4d568a654aedb750cc23" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36203 } }
I20260808 22:50:16.204152 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24210
I20260808 22:50:16.204180 24320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.204219 24320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dc2264f81c4d568a654aedb750cc23, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.204241 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:16.204416 22648 external_mini_cluster.cc:1428] Setting key 7abd03554675619ddbc683a5153ef82c
I20260808 22:50:16.204384 24320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dc2264f81c4d568a654aedb750cc23" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36203 } }
I20260808 22:50:16.204483 24320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:16.204516 24320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:16.204562 24320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:16.205220 24320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dc2264f81c4d568a654aedb750cc23" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36203 } }
I20260808 22:50:16.205343 24320 leader_election.cc:302] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05dc2264f81c4d568a654aedb750cc23; no voters:
I20260808 22:50:16.205526 24320 leader_election.cc:288] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:16.205575 24329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:16.205714 24329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [term 1 LEADER]: Becoming Leader. State: Replica: 05dc2264f81c4d568a654aedb750cc23, State: Running, Role: LEADER
I20260808 22:50:16.205812 24320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:16.205830 24329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dc2264f81c4d568a654aedb750cc23" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36203 } }
I20260808 22:50:16.206171 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05dc2264f81c4d568a654aedb750cc23. Latest consensus state: current_term: 1 leader_uuid: "05dc2264f81c4d568a654aedb750cc23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dc2264f81c4d568a654aedb750cc23" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36203 } } }
I20260808 22:50:16.206187 24330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05dc2264f81c4d568a654aedb750cc23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05dc2264f81c4d568a654aedb750cc23" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36203 } } }
I20260808 22:50:16.206326 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:16.206426 24330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:16.206665 24341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:16.207230 24341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:16.208807 24341 catalog_manager.cc:1347] Generated new cluster ID: 7928026de20b4c21bd267c64790c7236
I20260808 22:50:16.208859 24341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:16.216349 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.204919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.22.30.62:36203)
0808 22:50:16.205102 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.205112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.205142 (+ 30us) server_negotiation.cc:407] Connection header received
0808 22:50:16.205238 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.205243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.205284 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.205430 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.206293 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.207841 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.208669 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.209248 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.209344 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.212029 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.212032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.212033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.212044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.214731 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.215124 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.215127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.215127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.215252 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.215481 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.215482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.215483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.215778 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.215937 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.216014 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.216151 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260808 22:50:16.220686 24341 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:16.221328 24341 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:16.225049 24341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: Generated new TSK 0
I20260808 22:50:16.225200 24341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:16.254693 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36203
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:16.330813 24358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:16.331010 24358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:16.331048 24358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:16.332125 24358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:16.332182 24358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:16.332213 24358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:16.332247 24358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:16.333487 24358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:16.333730 24358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:16.333945 24358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:16.338819 24358 server_base.cc:1034] running on GCE node
W20260808 22:50:16.338788 24367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:16.338972 24371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:16.338793 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:16.339188 24358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:16.339498 24358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:16.340665 24358 hybrid_clock.cc:648] HybridClock initialized: now 1786229416340652 us; error 25 us; skew 500 ppm
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:16.342936 24358 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:16.343420 24358 webserver.cc:460] Webserver started at http://127.22.30.1:42791/ using document root <none> and password file <none>
I20260808 22:50:16.343659 24358 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:16.343725 24358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:16.343865 24358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:16.344839 24358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "fdff77db082c4359809a96706ef26299"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "cca8daefc99ce0232bdbe8173819a10d"
server_key_iv: "114b30afb0da46d707f14eceada5534b"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.345183 24358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "fdff77db082c4359809a96706ef26299"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "cca8daefc99ce0232bdbe8173819a10d"
server_key_iv: "114b30afb0da46d707f14eceada5534b"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.346446 24358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:16.347258 24379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.347445 24358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:16.347561 24358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "fdff77db082c4359809a96706ef26299"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "cca8daefc99ce0232bdbe8173819a10d"
server_key_iv: "114b30afb0da46d707f14eceada5534b"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.347647 24358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:16.363130 24358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:16.364856 24358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:16.365031 24358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:16.365650 24358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:16.365701 24358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.365731 24358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:16.365748 24358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:16.372910 24358 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:43411
I20260808 22:50:16.372983 24549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:43411 every 8 connection(s)
I20260808 22:50:16.373230 24358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:16.374133 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.366089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:55281 (local address 127.22.30.62:36203)
0808 22:50:16.366157 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.366160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.366321 (+ 161us) server_negotiation.cc:407] Connection header received
0808 22:50:16.366822 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.366823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.366839 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.366888 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.367555 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.368044 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.369011 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.369245 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.369334 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.371309 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.371310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.371311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.371317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.372753 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.373202 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.373203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.373204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.373262 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.373544 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.373545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.373545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.373676 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.373728 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.373975 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.374053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:16.374291 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.366044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.22.30.1:55281)
0808 22:50:16.366204 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.366241 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.366682 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.366965 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.366968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.367227 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:16.367462 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.367467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.368147 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.368148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.368916 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.368918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.369344 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.369345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.369496 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.369919 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:16.369933 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:16.371213 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:16.372833 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.372837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.372839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.373108 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.373328 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.373329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.373329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.373455 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.373730 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:16.373736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:16.373855 (+ 119us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.373953 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.374085 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260808 22:50:16.374761 24552 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36203
I20260808 22:50:16.374842 24552 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:16.375010 24552 heartbeater.cc:511] Master 127.22.30.62:36203 requested a full tablet report, sending...
I20260808 22:50:16.375501 24253 ts_manager.cc:194] Registered new tserver with Master: fdff77db082c4359809a96706ef26299 (127.22.30.1:43411)
I20260808 22:50:16.376399 24253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:55281
I20260808 22:50:16.379457 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24358
I20260808 22:50:16.379573 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:16.379745 22648 external_mini_cluster.cc:1428] Setting key e682f0c5e3b6ca0901f1c23d12338b27
I20260808 22:50:16.380527 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.375843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:41067 (local address 127.22.30.62:36203)
0808 22:50:16.375882 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.375883 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.375966 (+ 83us) server_negotiation.cc:407] Connection header received
0808 22:50:16.376025 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.376026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.376033 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.376074 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.376390 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.376927 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.377562 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.377705 (+ 143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.377744 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.378470 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.378471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.378471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.378475 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.379458 (+ 983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.379820 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.379821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.379822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.379871 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.380057 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.380058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.380058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.380152 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.380236 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.380384 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.380452 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:16.380533 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.375762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.22.30.1:41067)
0808 22:50:16.375904 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.375914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.375974 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.376079 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.376080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.376260 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:16.376324 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.376325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.377009 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.377010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.377501 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.377502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.377757 (+ 255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.377757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.377807 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.378168 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:16.378172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:16.378412 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:16.379534 (+ 1122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.379535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.379535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.379730 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.379938 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.379940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.379940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.379993 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.380230 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:16.380231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:16.380279 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.380360 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.380451 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:16.401625 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36203
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:16.475308 24564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:16.475517 24564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:16.475556 24564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:16.476816 24564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:16.476886 24564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:16.476925 24564 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:16.476967 24564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:16.478542 24564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:16.478698 24564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:16.478845 24564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:16.483346 24573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:16.483460 24564 server_base.cc:1034] running on GCE node
W20260808 22:50:16.483346 24570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:16.483349 24571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:16.483708 24564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:16.483947 24564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:16.485121 24564 hybrid_clock.cc:648] HybridClock initialized: now 1786229416485086 us; error 31 us; skew 500 ppm
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:16.487603 24564 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:16.488042 24564 webserver.cc:460] Webserver started at http://127.22.30.2:39825/ using document root <none> and password file <none>
I20260808 22:50:16.488236 24564 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:16.488303 24564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:16.488440 24564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:16.489324 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "cfdf876adbd04cb28c55a54e5a923182"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "ea3a75cd13f906e3145bf4e735a97226"
server_key_iv: "64645ff80968b63f3883fe7f9a8f40f0"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.489647 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "cfdf876adbd04cb28c55a54e5a923182"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "ea3a75cd13f906e3145bf4e735a97226"
server_key_iv: "64645ff80968b63f3883fe7f9a8f40f0"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.490906 24564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:16.491631 24581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.491825 24564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:16.491925 24564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "cfdf876adbd04cb28c55a54e5a923182"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "ea3a75cd13f906e3145bf4e735a97226"
server_key_iv: "64645ff80968b63f3883fe7f9a8f40f0"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.491994 24564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:16.500885 24564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:16.503041 24564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:16.503194 24564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:16.503743 24564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:16.503800 24564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.503831 24564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:16.503860 24564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:16.510941 24564 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:46007
I20260808 22:50:16.511135 24750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:46007 every 8 connection(s)
I20260808 22:50:16.511247 24564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:16.513371 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.504089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:47143 (local address 127.22.30.62:36203)
0808 22:50:16.504151 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.504153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.504395 (+ 242us) server_negotiation.cc:407] Connection header received
0808 22:50:16.504855 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.504857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.504869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.504932 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.505594 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.506381 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.508075 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.508293 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.508382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.510653 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.510655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.510655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.510661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.512040 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.512492 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.512494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.512494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.512549 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.512799 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.512800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.512800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.512906 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.512974 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.513220 (+ 246us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.513293 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 22:50:16.513532 24594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.504097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.22.30.2:47143)
0808 22:50:16.504250 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.504285 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.504753 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.505001 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.505004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.505296 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:16.505503 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.505508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.506483 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.506485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.507972 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.507976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.508404 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.508405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.508558 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.509052 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:16.509062 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:16.510550 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:16.512133 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.512140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.512142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.512407 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.512618 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.512619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.512619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.512721 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.512968 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:16.512974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:16.513095 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.513186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.513328 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260808 22:50:16.514014 24751 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36203
I20260808 22:50:16.514091 24751 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:16.514248 24751 heartbeater.cc:511] Master 127.22.30.62:36203 requested a full tablet report, sending...
I20260808 22:50:16.514565 24252 ts_manager.cc:194] Registered new tserver with Master: cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007)
I20260808 22:50:16.514919 24252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:47143
I20260808 22:50:16.515548 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24564
I20260808 22:50:16.515645 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:16.515841 22648 external_mini_cluster.cc:1428] Setting key c0105fe739d32cc93e71decd1f83580c
I20260808 22:50:16.520820 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.514992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:53867 (local address 127.22.30.62:36203)
0808 22:50:16.515082 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.515084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.515194 (+ 110us) server_negotiation.cc:407] Connection header received
0808 22:50:16.515248 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.515249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.515257 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.515324 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.515776 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.516499 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.517164 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.517343 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.517408 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.518276 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.518278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.518278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.518284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.519636 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.520018 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.520019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.520019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.520064 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.520284 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.520285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.520286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.520417 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.520464 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.520703 (+ 239us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.520758 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":73}
I20260808 22:50:16.520884 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.514952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.22.30.2:53867)
0808 22:50:16.515134 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.515142 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.515204 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.515337 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.515338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.515561 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:16.515655 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.515657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.516594 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.516595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.517068 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.517070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.517409 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.517410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.517477 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.517861 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:16.517866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:16.518206 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:16.519709 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.519710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.519710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.519934 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.520126 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.520127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.520127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.520196 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.520485 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:16.520486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:16.520535 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.520624 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.520729 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:16.539917 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36203
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:16.615761 24762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:16.615935 24762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:16.615958 24762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:16.617051 24762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:16.617084 24762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:16.617125 24762 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:16.617156 24762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:16.618424 24762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:16.618635 24762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:16.618840 24762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:16.623348 24772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:16.623363 24774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:16.623363 24769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:16.623723 24762 server_base.cc:1034] running on GCE node
I20260808 22:50:16.623957 24762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:16.624169 24762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:16.625311 24762 hybrid_clock.cc:648] HybridClock initialized: now 1786229416625297 us; error 37 us; skew 500 ppm
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:16.627604 24762 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:16.628023 24762 webserver.cc:460] Webserver started at http://127.22.30.3:40557/ using document root <none> and password file <none>
I20260808 22:50:16.628211 24762 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:16.628247 24762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:16.628346 24762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:16.629258 24762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "0ae2957862bc444cb193459d36dfc81f"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "3cce1b4c8a89ac92d3f2671e59b8eda0"
server_key_iv: "5b90293a537e81988b70b2216154cbfd"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.629549 24762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "0ae2957862bc444cb193459d36dfc81f"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "3cce1b4c8a89ac92d3f2671e59b8eda0"
server_key_iv: "5b90293a537e81988b70b2216154cbfd"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.630715 24762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:16.631309 24784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.631481 24762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.631546 24762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "0ae2957862bc444cb193459d36dfc81f"
format_stamp: "Formatted at 2026-08-08 22:50:16 on dist-test-slave-x4qh"
server_key: "3cce1b4c8a89ac92d3f2671e59b8eda0"
server_key_iv: "5b90293a537e81988b70b2216154cbfd"
server_key_version: "encryptionkey@0"
I20260808 22:50:16.631632 24762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:16.646508 24762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:16.648335 24762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:16.648480 24762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:16.648980 24762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:16.649017 24762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:16.649039 24762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:16.649082 24762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:16.657284 24762 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:32957
I20260808 22:50:16.657382 24954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:32957 every 8 connection(s)
I20260808 22:50:16.657666 24762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:16.658882 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.649326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57135 (local address 127.22.30.62:36203)
0808 22:50:16.649365 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.649367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.649548 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:50:16.650018 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.650019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.650032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.650087 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.650692 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.651188 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.652620 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.652880 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.652964 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.656199 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.656200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.656201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.656208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.657755 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.658206 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.658208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.658208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.658270 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.658469 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.658469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.658469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.658562 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.658608 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.658771 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.658827 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:16.659085 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.649310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.22.30.3:57135)
0808 22:50:16.649446 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.649476 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.649913 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.650147 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.650150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.650421 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:16.650596 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.650601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.651295 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.651296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.652526 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.652528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.652976 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.652977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.653127 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.653631 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:16.653641 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:16.656103 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:16.657838 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.657850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.657852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.658122 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.658325 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.658326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.658327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.658411 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.658600 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:16.658602 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:16.658692 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.658759 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.658865 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260808 22:50:16.659474 24955 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36203
I20260808 22:50:16.659552 24955 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:16.659723 24955 heartbeater.cc:511] Master 127.22.30.62:36203 requested a full tablet report, sending...
I20260808 22:50:16.660064 24253 ts_manager.cc:194] Registered new tserver with Master: 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957)
I20260808 22:50:16.660420 24253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:57135
I20260808 22:50:16.664444 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24762
I20260808 22:50:16.664543 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:16.664732 22648 external_mini_cluster.cc:1428] Setting key 16e43166a0a386b8f9d84d347392c78a
I20260808 22:50:16.665700 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:50:16.666033 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.660453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.22.30.3:50557)
0808 22:50:16.660570 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.660580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.660637 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.661129 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.661129 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.661328 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:16.661419 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.661420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.661943 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.661944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.662381 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.662382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.662656 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.662657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.662708 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.663021 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:16.663024 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:16.663250 (+ 226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:16.664953 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.664954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.664955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.665171 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.665390 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:16.665391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:16.665391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:16.665452 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:16.665743 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:16.665743 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:16.665820 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.665898 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.665973 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260808 22:50:16.666072 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.660494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:50557 (local address 127.22.30.62:36203)
0808 22:50:16.660575 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.660576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.660625 (+ 49us) server_negotiation.cc:407] Connection header received
0808 22:50:16.660684 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.660685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.660691 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.660736 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.661479 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.661866 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.662439 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.662590 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.662633 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.663318 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.663319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.663320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.663325 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.664876 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.665239 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.665240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.665241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.665311 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.665526 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.665527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.665528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.665639 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.665722 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.665930 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.665999 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:50:16 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:16.688827 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.681983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.22.30.62:36203)
0808 22:50:16.682036 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.682038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.682046 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:16.682140 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.682141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.682152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.682211 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:16.682755 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.683186 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.683922 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.684089 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.684176 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.686232 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:16.686234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:16.686234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:16.686241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:16.687762 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.688106 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.688107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.688108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.688159 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:16.688370 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:16.688371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:16.688372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:16.688474 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:16.688572 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.688676 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.688744 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:50:16.690201 24253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:16.691815 24253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:16.697952 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.693993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:32957 (local address 127.0.0.1:51590)
0808 22:50:16.694193 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.694203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.694258 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.694553 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.694554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.694556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:16.694589 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.694590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.695814 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.695815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.696786 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.696789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.697672 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.697752 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.697765 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.697836 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.697890 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260808 22:50:16.698053 24980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.693810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43411 (local address 127.0.0.1:47980)
0808 22:50:16.694127 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.694136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.694177 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.694537 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.694538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.694540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:16.694655 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.694658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.695621 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.695622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.697043 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.697045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.697764 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.697824 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.697828 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.697894 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.697937 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":82,"thread_start_us":64,"threads_started":1}
I20260808 22:50:16.698189 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.693784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:46007 (local address 127.0.0.1:37650)
0808 22:50:16.694036 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.694070 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.694141 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.694439 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.694443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.694446 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:16.694517 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.694518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.695796 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.695797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.697042 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.697045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.697920 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.697977 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.697981 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.698031 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.698068 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260808 22:50:16.698180 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.694193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.22.30.3:32957)
0808 22:50:16.694356 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.694360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.694369 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:16.694405 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.694406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.694422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.694502 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:16.694660 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.695694 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.696885 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.697583 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697653 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.697703 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.698063 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.698099 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260808 22:50:16.698251 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.693933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.22.30.1:43411)
0808 22:50:16.694328 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.694333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.694343 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:16.694380 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.694382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.694397 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.694473 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:16.694739 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.695528 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697138 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.697694 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697760 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.697810 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.698102 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.698150 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":159,"threads_started":1}
I20260808 22:50:16.698475 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.693874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37650 (local address 127.22.30.2:46007)
0808 22:50:16.694014 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.694018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.694228 (+ 210us) server_negotiation.cc:407] Connection header received
0808 22:50:16.694287 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.694319 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.694338 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.694423 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:16.694600 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.695702 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697126 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.697858 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.697916 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.697963 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.698349 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.698391 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260808 22:50:16.698935 24451 tablet_service.cc:1467] Processing CreateTablet for tablet 53a999e512524e8c97f9082697efacf1 (DEFAULT_TABLE table=test-table [id=f396503ba7274b23b10687ca588a436a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:16.698992 24649 tablet_service.cc:1467] Processing CreateTablet for tablet 53a999e512524e8c97f9082697efacf1 (DEFAULT_TABLE table=test-table [id=f396503ba7274b23b10687ca588a436a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:16.698978 24863 tablet_service.cc:1467] Processing CreateTablet for tablet 53a999e512524e8c97f9082697efacf1 (DEFAULT_TABLE table=test-table [id=f396503ba7274b23b10687ca588a436a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:16.699277 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a999e512524e8c97f9082697efacf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.699285 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a999e512524e8c97f9082697efacf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.699295 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a999e512524e8c97f9082697efacf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.701938 24985 tablet_bootstrap.cc:492] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Bootstrap starting.
I20260808 22:50:16.702484 24986 tablet_bootstrap.cc:492] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Bootstrap starting.
I20260808 22:50:16.702607 24985 tablet_bootstrap.cc:654] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.702961 24985 log.cc:824] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:16.703265 24989 tablet_bootstrap.cc:492] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Bootstrap starting.
I20260808 22:50:16.703444 24986 tablet_bootstrap.cc:654] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.703748 24985 tablet_bootstrap.cc:492] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: No bootstrap required, opened a new log
I20260808 22:50:16.703810 24986 log.cc:824] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:16.703838 24985 ts_tablet_manager.cc:1397] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:16.703963 24989 tablet_bootstrap.cc:654] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.704422 24989 log.cc:824] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:16.704651 24986 tablet_bootstrap.cc:492] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: No bootstrap required, opened a new log
I20260808 22:50:16.704737 24986 ts_tablet_manager.cc:1397] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:16.705363 24989 tablet_bootstrap.cc:492] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: No bootstrap required, opened a new log
I20260808 22:50:16.705502 24989 ts_tablet_manager.cc:1397] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:16.705948 24985 raft_consensus.cc:348] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.706099 24985 raft_consensus.cc:374] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.706157 24985 raft_consensus.cc:729] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdff77db082c4359809a96706ef26299, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.706312 24985 consensus_queue.cc:260] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.706653 24985 ts_tablet_manager.cc:1428] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:50:16.706785 24552 heartbeater.cc:503] Master 127.22.30.62:36203 was elected leader, sending a full tablet report...
I20260808 22:50:16.706887 24986 raft_consensus.cc:348] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.707094 24986 raft_consensus.cc:374] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.707190 24986 raft_consensus.cc:729] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae2957862bc444cb193459d36dfc81f, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.707350 24986 consensus_queue.cc:260] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.707747 24986 ts_tablet_manager.cc:1428] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260808 22:50:16.707715 24989 raft_consensus.cc:348] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.707795 24955 heartbeater.cc:503] Master 127.22.30.62:36203 was elected leader, sending a full tablet report...
I20260808 22:50:16.707870 24989 raft_consensus.cc:374] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.707909 24989 raft_consensus.cc:729] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdf876adbd04cb28c55a54e5a923182, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.708016 24989 consensus_queue.cc:260] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.708292 24989 ts_tablet_manager.cc:1428] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 22:50:16.708292 24751 heartbeater.cc:503] Master 127.22.30.62:36203 was elected leader, sending a full tablet report...
W20260808 22:50:16.761803 24752 tablet.cc:2369] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:16.869525 24995 raft_consensus.cc:482] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:16.869634 24995 raft_consensus.cc:504] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.870007 24995 leader_election.cc:288] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007), fdff77db082c4359809a96706ef26299 (127.22.30.1:43411)
I20260808 22:50:16.873801 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.870272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:33669 (local address 127.22.30.1:43411)
0808 22:50:16.870312 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.870314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.870528 (+ 214us) server_negotiation.cc:407] Connection header received
0808 22:50:16.870553 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.870554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.870563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.870626 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:16.870746 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.871335 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.872856 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.873408 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.873494 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.873503 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.873694 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.873731 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:16.873799 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.870302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:55385 (local address 127.22.30.2:46007)
0808 22:50:16.870356 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.870358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.870370 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:50:16.870396 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.870397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.870407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.870458 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:16.870648 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.871336 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.872855 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.873414 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.873494 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.873503 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.873695 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.873732 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:50:16.873842 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.870211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:46007 (local address 127.22.30.3:55385)
0808 22:50:16.870259 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.870270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.870336 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.870490 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.870491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.870493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:16.870566 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.870568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.871452 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.871452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.872762 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.872765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.873503 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.873503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.873614 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.873620 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.873676 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.873722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":79}
W20260808 22:50:16.874007 24553 tablet.cc:2369] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:16.873847 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.870213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43411 (local address 127.22.30.3:33669)
0808 22:50:16.870434 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:16.870459 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:16.870509 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:16.870632 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:16.870633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:16.870634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:16.870690 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.870691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.871443 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.871444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.872762 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:16.872765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.873503 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.873503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:16.873614 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.873620 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:50:16.873676 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:50:16.873722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":117,"thread_start_us":107,"threads_started":1}
I20260808 22:50:16.874210 24684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a999e512524e8c97f9082697efacf1" candidate_uuid: "0ae2957862bc444cb193459d36dfc81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfdf876adbd04cb28c55a54e5a923182" is_pre_election: true
I20260808 22:50:16.874361 24684 raft_consensus.cc:2454] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae2957862bc444cb193459d36dfc81f in term 0.
I20260808 22:50:16.874593 24478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a999e512524e8c97f9082697efacf1" candidate_uuid: "0ae2957862bc444cb193459d36dfc81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdff77db082c4359809a96706ef26299" is_pre_election: true
I20260808 22:50:16.874611 24789 leader_election.cc:302] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae2957862bc444cb193459d36dfc81f, cfdf876adbd04cb28c55a54e5a923182; no voters:
I20260808 22:50:16.874727 24478 raft_consensus.cc:2454] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae2957862bc444cb193459d36dfc81f in term 0.
I20260808 22:50:16.874837 24995 raft_consensus.cc:2781] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:16.874907 24995 raft_consensus.cc:482] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:16.874936 24995 raft_consensus.cc:3037] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:16.876039 24995 raft_consensus.cc:504] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.876186 24995 leader_election.cc:288] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [CANDIDATE]: Term 1 election: Requested vote from peers cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007), fdff77db082c4359809a96706ef26299 (127.22.30.1:43411)
I20260808 22:50:16.876328 24478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a999e512524e8c97f9082697efacf1" candidate_uuid: "0ae2957862bc444cb193459d36dfc81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdff77db082c4359809a96706ef26299"
I20260808 22:50:16.876340 24684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a999e512524e8c97f9082697efacf1" candidate_uuid: "0ae2957862bc444cb193459d36dfc81f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfdf876adbd04cb28c55a54e5a923182"
I20260808 22:50:16.876405 24478 raft_consensus.cc:3037] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:16.876442 24684 raft_consensus.cc:3037] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:16.877081 24478 raft_consensus.cc:2454] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae2957862bc444cb193459d36dfc81f in term 1.
I20260808 22:50:16.877261 24684 raft_consensus.cc:2454] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae2957862bc444cb193459d36dfc81f in term 1.
I20260808 22:50:16.877283 24785 leader_election.cc:302] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae2957862bc444cb193459d36dfc81f, fdff77db082c4359809a96706ef26299; no voters:
I20260808 22:50:16.877393 24995 raft_consensus.cc:2781] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:16.877514 24995 raft_consensus.cc:686] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 1 LEADER]: Becoming Leader. State: Replica: 0ae2957862bc444cb193459d36dfc81f, State: Running, Role: LEADER
I20260808 22:50:16.877614 24995 consensus_queue.cc:237] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.878398 24253 catalog_manager.cc:5495] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ae2957862bc444cb193459d36dfc81f (127.22.30.3). New cstate: current_term: 1 leader_uuid: "0ae2957862bc444cb193459d36dfc81f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } health_report { overall_health: HEALTHY } } }
W20260808 22:50:16.908290 24956 tablet.cc:2369] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:16.922019 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:16.920064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.22.30.3:32957)
0808 22:50:16.920111 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:16.920113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:16.920121 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:16.920185 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:16.920185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:16.920199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:16.920249 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:16.920383 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.920754 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.921406 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:16.921563 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:16.921606 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:16.921772 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:16.921848 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:16.921901 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:16.921945 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 22:50:16.921960 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:50:16.923506 24478 raft_consensus.cc:1261] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Refusing update from remote peer 0ae2957862bc444cb193459d36dfc81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:16.923506 24684 raft_consensus.cc:1261] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Refusing update from remote peer 0ae2957862bc444cb193459d36dfc81f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:16.923799 24995 consensus_queue.cc:1035] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:16.923959 25000 consensus_queue.cc:1035] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:16.926431 25004 mvcc.cc:205] Tried to move back new op lower bound from 7316395691716194304 to 7316395691531386880. Current Snapshot: MvccSnapshot[applied={T|T < 7316395691716194304}]
I20260808 22:50:16.930668 24253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:16.930795 24253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:16.931586 24253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: Sending DeleteTablet for 3 replicas of tablet 53a999e512524e8c97f9082697efacf1
I20260808 22:50:16.932017 24649 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a999e512524e8c97f9082697efacf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:16 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:37650
I20260808 22:50:16.932013 24451 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a999e512524e8c97f9082697efacf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:16 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47980
I20260808 22:50:16.932318 25016 tablet_replica.cc:330] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: stopping tablet replica
I20260808 22:50:16.932307 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a999e512524e8c97f9082697efacf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:16 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:51590
I20260808 22:50:16.932374 25017 tablet_replica.cc:330] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: stopping tablet replica
I20260808 22:50:16.932449 25016 raft_consensus.cc:2229] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:16.932499 25017 raft_consensus.cc:2229] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:16.932655 25018 tablet_replica.cc:330] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: stopping tablet replica
I20260808 22:50:16.932662 25017 raft_consensus.cc:2258] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:16.932803 25018 raft_consensus.cc:2229] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:16.932760 24253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:16.932900 24253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:16.933068 25018 raft_consensus.cc:2258] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:16.933112 25017 ts_tablet_manager.cc:1905] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:16.933642 25018 ts_tablet_manager.cc:1905] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:16.934542 24451 tablet_service.cc:1467] Processing CreateTablet for tablet 86efd4b325034df9a7642687d69a2367 (DEFAULT_TABLE table=test-table [id=4e9c5a9c720a4bb6862ada76d54a4070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:16.934583 24649 tablet_service.cc:1467] Processing CreateTablet for tablet 86efd4b325034df9a7642687d69a2367 (DEFAULT_TABLE table=test-table [id=4e9c5a9c720a4bb6862ada76d54a4070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:16.934655 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86efd4b325034df9a7642687d69a2367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.934691 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86efd4b325034df9a7642687d69a2367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.935956 25018 ts_tablet_manager.cc:1918] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:16.936015 25018 log.cc:1196] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/53a999e512524e8c97f9082697efacf1
I20260808 22:50:16.936317 24985 tablet_bootstrap.cc:492] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Bootstrap starting.
I20260808 22:50:16.936324 24863 tablet_service.cc:1467] Processing CreateTablet for tablet 86efd4b325034df9a7642687d69a2367 (DEFAULT_TABLE table=test-table [id=4e9c5a9c720a4bb6862ada76d54a4070]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:16.936421 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86efd4b325034df9a7642687d69a2367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:16.936576 24989 tablet_bootstrap.cc:492] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Bootstrap starting.
I20260808 22:50:16.936870 24985 tablet_bootstrap.cc:654] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.937075 25016 raft_consensus.cc:2258] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:16.937599 24989 tablet_bootstrap.cc:654] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.937820 25018 ts_tablet_manager.cc:1939] T 53a999e512524e8c97f9082697efacf1 P 0ae2957862bc444cb193459d36dfc81f: Deleting consensus metadata
I20260808 22:50:16.937987 25016 ts_tablet_manager.cc:1905] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:16.938184 24232 catalog_manager.cc:4841] TS 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957): tablet 53a999e512524e8c97f9082697efacf1 (table test-table [id=f396503ba7274b23b10687ca588a436a]) successfully deleted
I20260808 22:50:16.938475 24986 tablet_bootstrap.cc:492] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Bootstrap starting.
I20260808 22:50:16.939174 24986 tablet_bootstrap.cc:654] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:16.939730 24989 tablet_bootstrap.cc:492] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: No bootstrap required, opened a new log
I20260808 22:50:16.939790 24989 ts_tablet_manager.cc:1397] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260808 22:50:16.939855 25016 ts_tablet_manager.cc:1918] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:16.939970 25016 log.cc:1196] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/53a999e512524e8c97f9082697efacf1
I20260808 22:50:16.940136 24989 raft_consensus.cc:348] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.940217 24989 raft_consensus.cc:374] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.940258 24989 raft_consensus.cc:729] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdf876adbd04cb28c55a54e5a923182, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.940282 25016 ts_tablet_manager.cc:1939] T 53a999e512524e8c97f9082697efacf1 P fdff77db082c4359809a96706ef26299: Deleting consensus metadata
I20260808 22:50:16.940308 24989 consensus_queue.cc:260] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.940409 24989 ts_tablet_manager.cc:1428] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:16.940652 24232 catalog_manager.cc:4841] TS fdff77db082c4359809a96706ef26299 (127.22.30.1:43411): tablet 53a999e512524e8c97f9082697efacf1 (table test-table [id=f396503ba7274b23b10687ca588a436a]) successfully deleted
I20260808 22:50:16.940856 24985 tablet_bootstrap.cc:492] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: No bootstrap required, opened a new log
I20260808 22:50:16.940900 24985 ts_tablet_manager.cc:1397] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 22:50:16.941205 24985 raft_consensus.cc:348] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.941308 24985 raft_consensus.cc:374] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.941339 24985 raft_consensus.cc:729] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdff77db082c4359809a96706ef26299, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.941359 25017 ts_tablet_manager.cc:1918] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:16.941381 24985 consensus_queue.cc:260] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.941437 25017 log.cc:1196] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/53a999e512524e8c97f9082697efacf1
I20260808 22:50:16.941495 24985 ts_tablet_manager.cc:1428] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:16.941732 25017 ts_tablet_manager.cc:1939] T 53a999e512524e8c97f9082697efacf1 P cfdf876adbd04cb28c55a54e5a923182: Deleting consensus metadata
I20260808 22:50:16.942144 24238 catalog_manager.cc:4841] TS cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007): tablet 53a999e512524e8c97f9082697efacf1 (table test-table [id=f396503ba7274b23b10687ca588a436a]) successfully deleted
I20260808 22:50:16.942426 24986 tablet_bootstrap.cc:492] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: No bootstrap required, opened a new log
I20260808 22:50:16.942489 24986 ts_tablet_manager.cc:1397] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260808 22:50:16.942795 24986 raft_consensus.cc:348] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.942873 24986 raft_consensus.cc:374] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:16.942898 24986 raft_consensus.cc:729] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae2957862bc444cb193459d36dfc81f, State: Initialized, Role: FOLLOWER
I20260808 22:50:16.942943 24986 consensus_queue.cc:260] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:16.943045 24986 ts_tablet_manager.cc:1428] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:17.188437 24993 raft_consensus.cc:482] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:17.188541 24993 raft_consensus.cc:504] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.188912 24993 leader_election.cc:288] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007), 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957)
I20260808 22:50:17.191670 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.189241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:59847 (local address 127.22.30.2:46007)
0808 22:50:17.189282 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.189284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.189292 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:17.189352 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.189353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.189360 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.189420 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:17.189576 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.190024 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.190990 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.191310 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.191402 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.191411 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.191555 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.191587 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:17.191699 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.189138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:36211 (local address 127.22.30.3:32957)
0808 22:50:17.189185 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.189187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.189273 (+ 86us) server_negotiation.cc:407] Connection header received
0808 22:50:17.189370 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.189371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.189378 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.189439 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:17.189602 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.190024 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.190987 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.191319 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.191413 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.191433 (+ 20us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.191586 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.191619 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:50:17.191707 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.189069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:46007 (local address 127.22.30.1:59847)
0808 22:50:17.189202 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.189214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.189280 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.189425 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.189426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.189428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:17.189506 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.189508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.190108 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.190109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.190899 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.190901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.191409 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.191410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.191487 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.191490 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.191550 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.191586 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":40,"thread_start_us":55,"threads_started":1}
I20260808 22:50:17.191720 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.189066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:32957 (local address 127.22.30.1:36211)
0808 22:50:17.189188 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.189199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.189293 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.189442 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.189443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.189446 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:17.189534 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.189535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.190109 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.190110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.190895 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.190897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.191453 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.191454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.191518 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.191521 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.191581 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.191617 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":49,"thread_start_us":64,"threads_started":1}
I20260808 22:50:17.191984 24684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86efd4b325034df9a7642687d69a2367" candidate_uuid: "fdff77db082c4359809a96706ef26299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfdf876adbd04cb28c55a54e5a923182" is_pre_election: true
I20260808 22:50:17.192090 24684 raft_consensus.cc:2454] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdff77db082c4359809a96706ef26299 in term 0.
I20260808 22:50:17.192040 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86efd4b325034df9a7642687d69a2367" candidate_uuid: "fdff77db082c4359809a96706ef26299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae2957862bc444cb193459d36dfc81f" is_pre_election: true
I20260808 22:50:17.192186 24896 raft_consensus.cc:2454] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdff77db082c4359809a96706ef26299 in term 0.
I20260808 22:50:17.192250 24387 leader_election.cc:302] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfdf876adbd04cb28c55a54e5a923182, fdff77db082c4359809a96706ef26299; no voters:
I20260808 22:50:17.192401 24993 raft_consensus.cc:2781] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:17.192449 24993 raft_consensus.cc:482] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:17.192463 24993 raft_consensus.cc:3037] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:17.193500 24993 raft_consensus.cc:504] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.193635 24993 leader_election.cc:288] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [CANDIDATE]: Term 1 election: Requested vote from peers cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007), 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957)
I20260808 22:50:17.193785 24684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86efd4b325034df9a7642687d69a2367" candidate_uuid: "fdff77db082c4359809a96706ef26299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfdf876adbd04cb28c55a54e5a923182"
I20260808 22:50:17.193828 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86efd4b325034df9a7642687d69a2367" candidate_uuid: "fdff77db082c4359809a96706ef26299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae2957862bc444cb193459d36dfc81f"
I20260808 22:50:17.193871 24684 raft_consensus.cc:3037] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:17.193892 24896 raft_consensus.cc:3037] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:17.194671 24684 raft_consensus.cc:2454] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdff77db082c4359809a96706ef26299 in term 1.
I20260808 22:50:17.194671 24896 raft_consensus.cc:2454] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdff77db082c4359809a96706ef26299 in term 1.
I20260808 22:50:17.194877 24381 leader_election.cc:302] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae2957862bc444cb193459d36dfc81f, fdff77db082c4359809a96706ef26299; no voters:
I20260808 22:50:17.194984 24993 raft_consensus.cc:2781] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:17.195139 24993 raft_consensus.cc:686] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 1 LEADER]: Becoming Leader. State: Replica: fdff77db082c4359809a96706ef26299, State: Running, Role: LEADER
I20260808 22:50:17.195214 24993 consensus_queue.cc:237] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.195925 24253 catalog_manager.cc:5495] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdff77db082c4359809a96706ef26299 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "fdff77db082c4359809a96706ef26299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } health_report { overall_health: UNKNOWN } } }
Aug 08 22:50:17 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:17.250455 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.243986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.22.30.62:36203)
0808 22:50:17.244138 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.244140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.244165 (+ 25us) server_negotiation.cc:407] Connection header received
0808 22:50:17.244236 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.244237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.244246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.244321 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:17.244805 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.245270 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.245911 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.246087 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.246134 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.248035 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:17.248036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:17.248036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:17.248042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:17.249492 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:17.249830 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:17.249830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:17.249831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:17.249887 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:17.250050 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:17.250050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:17.250051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:17.250143 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:17.250190 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.250311 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.250392 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260808 22:50:17.250644 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.243941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36203 (local address 127.0.0.1:55852)
0808 22:50:17.244104 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.244115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.244179 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.244338 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.244339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.244678 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:17.244734 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.244736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.245341 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.245342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.245843 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.245844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.246147 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.246147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.246184 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.246641 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:17.246645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:17.247962 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:17.249560 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:17.249567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:17.249568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:17.249758 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:17.249946 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:17.249947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:17.249947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:17.249998 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:17.250197 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:17.250203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:17.250242 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.250296 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.250459 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":78,"thread_start_us":87,"threads_started":1}
I20260808 22:50:17.252521 24253 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:55852:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 22:50:17.252781 24253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:17.254217 24451 tablet_service.cc:1467] Processing CreateTablet for tablet bba7c566a1e54c9484d0c05cf45f697e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43750cd4eda5455cbcf297873affb237]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:17.254252 24863 tablet_service.cc:1467] Processing CreateTablet for tablet bba7c566a1e54c9484d0c05cf45f697e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43750cd4eda5455cbcf297873affb237]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:17.254225 24649 tablet_service.cc:1467] Processing CreateTablet for tablet bba7c566a1e54c9484d0c05cf45f697e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43750cd4eda5455cbcf297873affb237]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:17.254437 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba7c566a1e54c9484d0c05cf45f697e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:17.254437 24649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba7c566a1e54c9484d0c05cf45f697e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:17.254448 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba7c566a1e54c9484d0c05cf45f697e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:17.255707 24985 tablet_bootstrap.cc:492] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: Bootstrap starting.
I20260808 22:50:17.255908 24989 tablet_bootstrap.cc:492] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: Bootstrap starting.
I20260808 22:50:17.255915 24986 tablet_bootstrap.cc:492] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: Bootstrap starting.
I20260808 22:50:17.256232 24986 tablet_bootstrap.cc:654] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:17.256234 24985 tablet_bootstrap.cc:654] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:17.256363 24989 tablet_bootstrap.cc:654] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:17.256886 24986 tablet_bootstrap.cc:492] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: No bootstrap required, opened a new log
I20260808 22:50:17.256937 24986 ts_tablet_manager.cc:1397] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:17.257035 24986 raft_consensus.cc:348] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.257076 24986 raft_consensus.cc:374] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:17.257088 24986 raft_consensus.cc:729] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae2957862bc444cb193459d36dfc81f, State: Initialized, Role: FOLLOWER
I20260808 22:50:17.257118 24986 consensus_queue.cc:260] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.257205 24986 ts_tablet_manager.cc:1428] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:17.257233 24985 tablet_bootstrap.cc:492] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: No bootstrap required, opened a new log
I20260808 22:50:17.257273 24985 ts_tablet_manager.cc:1397] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:17.257237 25011 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.257323 25011 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.257404 24985 raft_consensus.cc:348] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.257493 24985 raft_consensus.cc:374] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:17.257524 24985 raft_consensus.cc:729] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdff77db082c4359809a96706ef26299, State: Initialized, Role: FOLLOWER
I20260808 22:50:17.257570 24985 consensus_queue.cc:260] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.257654 24985 ts_tablet_manager.cc:1428] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:17.257661 24989 tablet_bootstrap.cc:492] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: No bootstrap required, opened a new log
I20260808 22:50:17.257704 24989 ts_tablet_manager.cc:1397] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260808 22:50:17.257726 24993 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.257810 24993 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.257853 24989 raft_consensus.cc:348] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.257915 24989 raft_consensus.cc:374] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:17.257958 24989 raft_consensus.cc:729] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdf876adbd04cb28c55a54e5a923182, State: Initialized, Role: FOLLOWER
I20260808 22:50:17.258002 24989 consensus_queue.cc:260] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.258100 24989 ts_tablet_manager.cc:1428] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:17.258150 24996 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.258281 24996 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.373639 24996 raft_consensus.cc:482] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:17.373746 24996 raft_consensus.cc:504] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.374111 24996 leader_election.cc:288] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdff77db082c4359809a96706ef26299 (127.22.30.1:43411), 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957)
I20260808 22:50:17.377980 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.374623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:49445 (local address 127.22.30.3:32957)
0808 22:50:17.374673 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.374675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.374823 (+ 148us) server_negotiation.cc:407] Connection header received
0808 22:50:17.374933 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.374934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.374946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.375000 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:17.375124 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.375835 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.377079 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.377597 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.377671 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.377681 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.377885 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.377922 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 22:50:17.377981 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.374421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:43579 (local address 127.22.30.1:43411)
0808 22:50:17.374675 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.374677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.374687 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:17.374711 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.374711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.374720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.374782 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:17.374947 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.375572 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.377079 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.377611 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.377671 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.377681 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.377885 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.377920 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":49,"threads_started":1}
I20260808 22:50:17.378062 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.374322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43411 (local address 127.22.30.2:43579)
0808 22:50:17.374527 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.374537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.374590 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.374807 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.374808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.374810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:17.374879 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.374881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.375647 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.375649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.376983 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.376986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.377707 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.377708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.377793 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.377798 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.377853 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.377897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":22,"thread_start_us":72,"threads_started":1}
I20260808 22:50:17.378144 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.374487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:32957 (local address 127.22.30.2:49445)
0808 22:50:17.374760 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.374780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.374861 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.374998 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.374999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.375001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:17.375048 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.375049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.375910 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.375911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.376994 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.376996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.377707 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.377708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.377796 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.377799 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.377853 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.377897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":4,"thread_start_us":53,"threads_started":1}
I20260808 22:50:17.378374 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba7c566a1e54c9484d0c05cf45f697e" candidate_uuid: "cfdf876adbd04cb28c55a54e5a923182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae2957862bc444cb193459d36dfc81f" is_pre_election: true
I20260808 22:50:17.378387 24478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba7c566a1e54c9484d0c05cf45f697e" candidate_uuid: "cfdf876adbd04cb28c55a54e5a923182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdff77db082c4359809a96706ef26299" is_pre_election: true
I20260808 22:50:17.378465 24896 raft_consensus.cc:2454] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfdf876adbd04cb28c55a54e5a923182 in term 0.
I20260808 22:50:17.378482 24478 raft_consensus.cc:2454] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfdf876adbd04cb28c55a54e5a923182 in term 0.
I20260808 22:50:17.378651 24582 leader_election.cc:302] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae2957862bc444cb193459d36dfc81f, cfdf876adbd04cb28c55a54e5a923182; no voters:
I20260808 22:50:17.378773 24996 raft_consensus.cc:2781] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:17.378865 24996 raft_consensus.cc:482] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:17.378898 24996 raft_consensus.cc:3037] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:17.379760 24996 raft_consensus.cc:504] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.379891 24996 leader_election.cc:288] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [CANDIDATE]: Term 1 election: Requested vote from peers fdff77db082c4359809a96706ef26299 (127.22.30.1:43411), 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957)
I20260808 22:50:17.380172 24478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba7c566a1e54c9484d0c05cf45f697e" candidate_uuid: "cfdf876adbd04cb28c55a54e5a923182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdff77db082c4359809a96706ef26299"
I20260808 22:50:17.380172 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba7c566a1e54c9484d0c05cf45f697e" candidate_uuid: "cfdf876adbd04cb28c55a54e5a923182" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae2957862bc444cb193459d36dfc81f"
I20260808 22:50:17.380249 24896 raft_consensus.cc:3037] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:17.380247 24478 raft_consensus.cc:3037] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:17.380919 24478 raft_consensus.cc:2454] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfdf876adbd04cb28c55a54e5a923182 in term 1.
I20260808 22:50:17.381044 24896 raft_consensus.cc:2454] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfdf876adbd04cb28c55a54e5a923182 in term 1.
I20260808 22:50:17.381060 24582 leader_election.cc:302] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfdf876adbd04cb28c55a54e5a923182, fdff77db082c4359809a96706ef26299; no voters:
I20260808 22:50:17.381139 24996 raft_consensus.cc:2781] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:17.381237 24996 raft_consensus.cc:686] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [term 1 LEADER]: Becoming Leader. State: Replica: cfdf876adbd04cb28c55a54e5a923182, State: Running, Role: LEADER
I20260808 22:50:17.381315 24996 consensus_queue.cc:237] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } }
I20260808 22:50:17.381611 25047 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: TxnStatusTablet state changed. Reason: New leader cfdf876adbd04cb28c55a54e5a923182. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.381704 25047 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:17.381865 25049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 22:50:17.381935 24252 catalog_manager.cc:5495] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:17.382027 25049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 22:50:17.403720 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.401445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.22.30.2:46007)
0808 22:50:17.401487 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.401488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.401497 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:17.401584 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.401585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.401599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.401659 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:17.401809 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.402228 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.402924 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.403152 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.403227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.403423 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.403511 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.403583 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.403643 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.403661 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:50:17.403774 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.401374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:46007 (local address 127.0.0.1:37660)
0808 22:50:17.401440 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.401450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.401495 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.401670 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.401671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.401673 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:17.401726 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.401728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.402317 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.402318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.402847 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.402848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.403228 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.403229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.403292 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.403306 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.403583 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.403595 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.403639 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.403688 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 22:50:17.404914 24478 raft_consensus.cc:1261] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Refusing update from remote peer cfdf876adbd04cb28c55a54e5a923182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:17.404999 24896 raft_consensus.cc:1261] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Refusing update from remote peer cfdf876adbd04cb28c55a54e5a923182: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:17.405234 25047 consensus_queue.cc:1035] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:17.405357 24996 consensus_queue.cc:1035] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:17.405944 24993 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: TxnStatusTablet state changed. Reason: New leader cfdf876adbd04cb28c55a54e5a923182. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.405979 25011 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: TxnStatusTablet state changed. Reason: New leader cfdf876adbd04cb28c55a54e5a923182. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.406052 24993 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.406064 25011 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.406894 25005 mvcc.cc:205] Tried to move back new op lower bound from 7316395693688283136 to 7316395693594202112. Current Snapshot: MvccSnapshot[applied={T|T < 7316395693688283136}]
I20260808 22:50:17.406975 25047 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.407038 25047 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:17.407212 25047 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.407294 25047 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P cfdf876adbd04cb28c55a54e5a923182: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:17.407567 24993 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.407598 25003 mvcc.cc:205] Tried to move back new op lower bound from 7316395693688283136 to 7316395693594202112. Current Snapshot: MvccSnapshot[applied={T|T < 7316395693688283136}]
I20260808 22:50:17.407634 24993 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P fdff77db082c4359809a96706ef26299: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.407821 25011 tablet_replica.cc:439] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cfdf876adbd04cb28c55a54e5a923182" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdff77db082c4359809a96706ef26299" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43411 } } peers { permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 } } peers { permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 } } }
I20260808 22:50:17.407924 25011 tablet_replica.cc:442] T bba7c566a1e54c9484d0c05cf45f697e P 0ae2957862bc444cb193459d36dfc81f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:17.412221 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.410055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.22.30.1:43411)
0808 22:50:17.410098 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.410100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.410109 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:17.410150 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.410151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.410166 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.410222 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:17.410323 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.410720 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.411389 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.411583 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.411654 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.411786 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.411878 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.411961 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.412021 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.412040 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:17.417515 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.415063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:53707 (local address 127.22.30.2:46007)
0808 22:50:17.415098 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.415099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.415163 (+ 64us) server_negotiation.cc:407] Connection header received
0808 22:50:17.415222 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.415222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.415228 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.415269 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:17.415477 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.415871 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.416955 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.417096 (+ 141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.417137 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.417366 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.417386 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.417418 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.417468 (+ 50us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.417480 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:17.417584 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.414841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:46007 (local address 127.22.30.1:53707)
0808 22:50:17.415102 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.415112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.415169 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.415272 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.415272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.415280 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:17.415425 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.415427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.415943 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.415944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.416870 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.416873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.417162 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.417164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.417281 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.417295 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.417421 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.417422 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.417464 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.417499 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
I20260808 22:50:17.419584 24896 raft_consensus.cc:1261] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Refusing update from remote peer fdff77db082c4359809a96706ef26299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:17.419616 24684 raft_consensus.cc:1261] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Refusing update from remote peer fdff77db082c4359809a96706ef26299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:17.419847 25026 consensus_queue.cc:1035] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae2957862bc444cb193459d36dfc81f" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:17.419961 24993 consensus_queue.cc:1035] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdf876adbd04cb28c55a54e5a923182" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:17.428858 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.425768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:42275 (local address 127.22.30.1:43411)
0808 22:50:17.425801 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.425802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.426078 (+ 276us) server_negotiation.cc:407] Connection header received
0808 22:50:17.426180 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.426181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.426189 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.426260 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:17.426466 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.427119 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.427971 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.428154 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.428225 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.428447 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.428478 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.428516 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.428776 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.428798 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:17.429116 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.425489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43411 (local address 127.22.30.2:42275)
0808 22:50:17.426010 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:17.426019 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:17.426090 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:17.426264 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:17.426264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:17.426273 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:17.426378 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.426381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.427189 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.427190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.427891 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:17.427893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.428234 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.428235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:17.428346 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.428357 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:17.428661 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:17.428662 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:17.428967 (+ 305us) client_negotiation.cc:240] Negotiation successful
0808 22:50:17.429008 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":142,"threads_started":1}
I20260808 22:50:17.438743 24253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:17.438838 24253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:17.439514 24253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05dc2264f81c4d568a654aedb750cc23: Sending DeleteTablet for 3 replicas of tablet 86efd4b325034df9a7642687d69a2367
I20260808 22:50:17.439740 24649 tablet_service.cc:1514] Processing DeleteTablet for tablet 86efd4b325034df9a7642687d69a2367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:17 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:37650
I20260808 22:50:17.439801 24451 tablet_service.cc:1514] Processing DeleteTablet for tablet 86efd4b325034df9a7642687d69a2367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:17 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47980
I20260808 22:50:17.439839 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet 86efd4b325034df9a7642687d69a2367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:17 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:51590
I20260808 22:50:17.439963 25017 tablet_replica.cc:330] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: stopping tablet replica
I20260808 22:50:17.439988 25070 tablet_replica.cc:330] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: stopping tablet replica
I20260808 22:50:17.440013 25017 raft_consensus.cc:2229] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:17.440037 25070 raft_consensus.cc:2229] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:17.440068 25070 raft_consensus.cc:2258] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:17.440062 25017 raft_consensus.cc:2258] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:17.440155 25016 tablet_replica.cc:330] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: stopping tablet replica
I20260808 22:50:17.440215 25016 raft_consensus.cc:2229] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:17.440290 25016 raft_consensus.cc:2258] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:17.440691 25016 ts_tablet_manager.cc:1905] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:17.440696 25017 ts_tablet_manager.cc:1905] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:17.442399 25016 ts_tablet_manager.cc:1918] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:17.442448 25016 log.cc:1196] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/86efd4b325034df9a7642687d69a2367
I20260808 22:50:17.442692 25016 ts_tablet_manager.cc:1939] T 86efd4b325034df9a7642687d69a2367 P fdff77db082c4359809a96706ef26299: Deleting consensus metadata
I20260808 22:50:17.442869 25017 ts_tablet_manager.cc:1918] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:17.442921 25017 log.cc:1196] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/86efd4b325034df9a7642687d69a2367
I20260808 22:50:17.442953 24232 catalog_manager.cc:4841] TS fdff77db082c4359809a96706ef26299 (127.22.30.1:43411): tablet 86efd4b325034df9a7642687d69a2367 (table test-table [id=4e9c5a9c720a4bb6862ada76d54a4070]) successfully deleted
I20260808 22:50:17.443114 25017 ts_tablet_manager.cc:1939] T 86efd4b325034df9a7642687d69a2367 P cfdf876adbd04cb28c55a54e5a923182: Deleting consensus metadata
I20260808 22:50:17.443475 24238 catalog_manager.cc:4841] TS cfdf876adbd04cb28c55a54e5a923182 (127.22.30.2:46007): tablet 86efd4b325034df9a7642687d69a2367 (table test-table [id=4e9c5a9c720a4bb6862ada76d54a4070]) successfully deleted
I20260808 22:50:17.447523 25070 ts_tablet_manager.cc:1905] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:17.448714 25070 ts_tablet_manager.cc:1918] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:17.448804 25070 log.cc:1196] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/86efd4b325034df9a7642687d69a2367
I20260808 22:50:17.449059 25070 ts_tablet_manager.cc:1939] T 86efd4b325034df9a7642687d69a2367 P 0ae2957862bc444cb193459d36dfc81f: Deleting consensus metadata
I20260808 22:50:17.449378 24232 catalog_manager.cc:4841] TS 0ae2957862bc444cb193459d36dfc81f (127.22.30.3:32957): tablet 86efd4b325034df9a7642687d69a2367 (table test-table [id=4e9c5a9c720a4bb6862ada76d54a4070]) successfully deleted
Aug 08 22:50:17 dist-test-slave-x4qh krb5kdc[24189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229416, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:17.461100 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.447616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.22.30.1:43411)
0808 22:50:17.447653 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.447655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.451523 (+ 3868us) server_negotiation.cc:407] Connection header received
0808 22:50:17.451569 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.451570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.451579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.451637 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:17.452390 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.453086 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.454134 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.454395 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.454471 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.457774 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:17.457781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:17.457781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:17.457787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:17.459799 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:17.460196 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:17.460199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:17.460199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:17.460279 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:17.460497 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:17.460498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:17.460498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:17.460702 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:17.460848 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.460940 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.461014 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 22:50:17.461664 24525 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47996
I20260808 22:50:17.470206 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:17.462743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.22.30.62:36203)
0808 22:50:17.462791 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:17.462794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:17.462966 (+ 172us) server_negotiation.cc:407] Connection header received
0808 22:50:17.463012 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:17.463012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:17.463023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:17.463076 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:17.463813 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.464567 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.465524 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:17.465848 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:17.465918 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:17.467321 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:17.467323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:17.467323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:17.467330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:17.468952 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:17.469334 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:17.469335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:17.469336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:17.469392 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:17.469601 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:17.469602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:17.469602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:17.469765 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:17.469864 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:17.470006 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 22:50:17.470109 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 22:50:17.470521 24253 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55854
I20260808 22:50:17.471149 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24358
I20260808 22:50:17.498188 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24564
I20260808 22:50:17.510108 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24762
I20260808 22:50:17.518997 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24210
2026-08-08T22:50:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2686 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:17 dist-test-slave-x4qh krb5kdc[25077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:17 dist-test-slave-x4qh krb5kdc[25077](info): set up 2 sockets
Aug 08 22:50:17 dist-test-slave-x4qh krb5kdc[25077](info): commencing operation
krb5kdc: starting...
W20260808 22:50:18.870663 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:18 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:18.975492 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:40589
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:40589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:19.049541 25096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:19.049716 25096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:19.049738 25096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:19.050784 25096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:19.050818 25096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 22:50:19.050832 25096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:19.050845 25096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:19.050858 25096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:19.050869 25096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:19.052186 25096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:40589
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:40589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:19.052345 25096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:19.052490 25096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:19.057065 25106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.057071 25103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.057072 25104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:19.057343 25096 server_base.cc:1034] running on GCE node
I20260808 22:50:19.057544 25096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:19.057843 25096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:19.059008 25096 hybrid_clock.cc:648] HybridClock initialized: now 1786229419058994 us; error 32 us; skew 500 ppm
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:19.066179 25096 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:19.066532 25096 webserver.cc:460] Webserver started at http://127.22.30.62:37207/ using document root <none> and password file <none>
I20260808 22:50:19.066725 25096 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:19.066763 25096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:19.066864 25096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:19.067829 25096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "93025b48668e4797bd2555e2368445bd"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "d49f7cc07cfc9d465799dd7f6d58d478"
server_key_iv: "94f0df9eea870a8b7ad081c94e965c11"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.068140 25096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "93025b48668e4797bd2555e2368445bd"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "d49f7cc07cfc9d465799dd7f6d58d478"
server_key_iv: "94f0df9eea870a8b7ad081c94e965c11"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.069293 25096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:19.069988 25120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.070149 25096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:19.070245 25096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "93025b48668e4797bd2555e2368445bd"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "d49f7cc07cfc9d465799dd7f6d58d478"
server_key_iv: "94f0df9eea870a8b7ad081c94e965c11"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.070348 25096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:19.090938 25096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:19.092592 25096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:19.092742 25096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:19.097069 25096 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:40589
I20260808 22:50:19.097136 25214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:40589 every 8 connection(s)
I20260808 22:50:19.097440 25096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:19.097967 25215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.099943 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25096
I20260808 22:50:19.099954 25215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Bootstrap starting.
I20260808 22:50:19.100041 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:19.100204 22648 external_mini_cluster.cc:1428] Setting key feb556ea56d6b76c7db3f7554772fe52
I20260808 22:50:19.100574 25215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.100905 25215 log.cc:824] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:19.101624 25215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: No bootstrap required, opened a new log
I20260808 22:50:19.103716 25215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93025b48668e4797bd2555e2368445bd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40589 } }
I20260808 22:50:19.103861 25215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.103902 25215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93025b48668e4797bd2555e2368445bd, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.104064 25215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93025b48668e4797bd2555e2368445bd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40589 } }
I20260808 22:50:19.104151 25215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:19.104194 25215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:19.104254 25215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.104873 25215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93025b48668e4797bd2555e2368445bd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40589 } }
I20260808 22:50:19.105006 25215 leader_election.cc:302] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93025b48668e4797bd2555e2368445bd; no voters:
I20260808 22:50:19.105167 25215 leader_election.cc:288] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:19.105193 25220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:19.105309 25220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [term 1 LEADER]: Becoming Leader. State: Replica: 93025b48668e4797bd2555e2368445bd, State: Running, Role: LEADER
I20260808 22:50:19.105387 25220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93025b48668e4797bd2555e2368445bd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40589 } }
I20260808 22:50:19.105417 25215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:19.105845 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93025b48668e4797bd2555e2368445bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93025b48668e4797bd2555e2368445bd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40589 } } }
I20260808 22:50:19.105969 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:19.105845 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93025b48668e4797bd2555e2368445bd. Latest consensus state: current_term: 1 leader_uuid: "93025b48668e4797bd2555e2368445bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93025b48668e4797bd2555e2368445bd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40589 } } }
I20260808 22:50:19.106120 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:19.106290 25230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:19.106781 25230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:19.108624 25230 catalog_manager.cc:1347] Generated new cluster ID: dd959b7926fa4159b409ad00575b4240
I20260808 22:50:19.108678 25230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:19.110502 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.100658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.22.30.62:40589)
0808 22:50:19.100838 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.100844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.100866 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:50:19.101269 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.101276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.101330 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.101434 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.102145 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.102917 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.103951 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.104527 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.104603 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.107404 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.107407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.107408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.107418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.109227 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.109544 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.109546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.109547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.109636 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.109813 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.109813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.109814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.110069 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.110191 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.110243 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.110351 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260808 22:50:19.121214 25230 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:19.121670 25230 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:19.126681 25230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Generated new TSK 0
I20260808 22:50:19.126868 25230 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:19.153631 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40589
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:19.229578 25255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:19.229759 25255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:19.229782 25255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:19.230819 25255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:19.230857 25255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:19.230875 25255 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 22:50:19.230886 25255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:19.230930 25255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:19.232235 25255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:40589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:19.232450 25255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:19.232689 25255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:19.237512 25261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.237504 25262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.237504 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:19.237725 25255 server_base.cc:1034] running on GCE node
I20260808 22:50:19.237929 25255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:19.238135 25255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:19.239287 25255 hybrid_clock.cc:648] HybridClock initialized: now 1786229419239269 us; error 34 us; skew 500 ppm
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:19.241576 25255 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:19.241984 25255 webserver.cc:460] Webserver started at http://127.22.30.1:34551/ using document root <none> and password file <none>
I20260808 22:50:19.242216 25255 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:19.242283 25255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:19.242422 25255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:19.243443 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "20a4da4311fb4fe3a0e10139a27d791b"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "e2f7112682d326f3baf377ebcc776a5a"
server_key_iv: "7192d8bd41b7022718aa8a1b812cc24f"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.243817 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "20a4da4311fb4fe3a0e10139a27d791b"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "e2f7112682d326f3baf377ebcc776a5a"
server_key_iv: "7192d8bd41b7022718aa8a1b812cc24f"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.245179 25255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:19.246058 25274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.246272 25255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:19.246363 25255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "20a4da4311fb4fe3a0e10139a27d791b"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "e2f7112682d326f3baf377ebcc776a5a"
server_key_iv: "7192d8bd41b7022718aa8a1b812cc24f"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.246439 25255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:19.270681 25255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:19.272385 25255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:19.272481 25255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:19.273020 25255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:19.273059 25255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.273078 25255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:19.273090 25255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:19.279789 25255 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:32897
I20260808 22:50:19.279840 25448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:32897 every 8 connection(s)
I20260808 22:50:19.280203 25255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:19.282516 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.273456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60479 (local address 127.22.30.62:40589)
0808 22:50:19.273535 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.273537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.273717 (+ 180us) server_negotiation.cc:407] Connection header received
0808 22:50:19.274185 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.274187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.274198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.274285 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.275027 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.275850 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.277188 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.277550 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.277625 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.280183 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.280185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.280185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.280192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.281454 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.281811 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.281812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.281812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.281862 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.282032 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.282033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.282033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.282127 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.282175 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.282392 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.282453 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 22:50:19.282689 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.273422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.22.30.1:60479)
0808 22:50:19.273594 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.273627 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.274090 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.274332 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.274335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.274667 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.274925 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.274931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.275982 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.275983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.277085 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.277087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.277664 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.277665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.277801 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.278294 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.278301 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.280076 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.281531 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.281536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.281538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.281744 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.281903 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.281903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.281903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.281981 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.282184 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.282189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.282292 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.282369 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.282476 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260808 22:50:19.283159 25449 heartbeater.cc:347] Connected to a master server at 127.22.30.62:40589
I20260808 22:50:19.283255 25449 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:19.283465 25449 heartbeater.cc:511] Master 127.22.30.62:40589 requested a full tablet report, sending...
I20260808 22:50:19.284026 25147 ts_manager.cc:194] Registered new tserver with Master: 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897)
I20260808 22:50:19.284843 25147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:60479
I20260808 22:50:19.288264 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25255
I20260808 22:50:19.288369 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:19.288568 22648 external_mini_cluster.cc:1428] Setting key c8dd3b0ca8f90cd990d95dc1e65d4070
I20260808 22:50:19.289324 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.284172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:54375 (local address 127.22.30.62:40589)
0808 22:50:19.284206 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.284207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.284360 (+ 153us) server_negotiation.cc:407] Connection header received
0808 22:50:19.284389 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.284390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.284399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.284462 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.284703 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.285347 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.286020 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.286187 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.286240 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.287026 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.287027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.287027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.287031 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.288180 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.288546 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.288547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.288547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.288604 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.288840 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.288841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.288841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.288961 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.289011 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.289159 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.289213 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:19.289327 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.284110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.22.30.1:54375)
0808 22:50:19.284269 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.284284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.284356 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.284466 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.284466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.284601 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.284644 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.284645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.285440 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.285441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.285947 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.285948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.286251 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.286252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.286308 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.286700 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.286711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.286957 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.288255 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.288256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.288256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.288467 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.288692 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.288693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.288693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.288764 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.289026 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.289027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.289069 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.289144 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.289237 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:19.311614 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40589
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:19.387485 25465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:19.387660 25465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:19.387683 25465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:19.388742 25465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:19.388797 25465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:19.388816 25465 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 22:50:19.388827 25465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:19.388844 25465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:19.390239 25465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:40589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:19.390424 25465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:19.390583 25465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:19.395190 25473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.395196 25474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:19.395455 25465 server_base.cc:1034] running on GCE node
W20260808 22:50:19.395190 25478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:19.395694 25465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:19.395936 25465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:19.397092 25465 hybrid_clock.cc:648] HybridClock initialized: now 1786229419397088 us; error 36 us; skew 500 ppm
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:19.399498 25465 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:19.399873 25465 webserver.cc:460] Webserver started at http://127.22.30.2:45939/ using document root <none> and password file <none>
I20260808 22:50:19.400080 25465 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:19.400141 25465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:19.400261 25465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:19.401232 25465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "4108386c6c1d44bba84a68d542b52b3f"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "72309d6b4c269aa93d55ecc6f477244c"
server_key_iv: "70f48438d39a81004086bb7ca83578f8"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.401587 25465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "4108386c6c1d44bba84a68d542b52b3f"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "72309d6b4c269aa93d55ecc6f477244c"
server_key_iv: "70f48438d39a81004086bb7ca83578f8"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.402801 25465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:19.403455 25488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.403623 25465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:19.403717 25465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "4108386c6c1d44bba84a68d542b52b3f"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "72309d6b4c269aa93d55ecc6f477244c"
server_key_iv: "70f48438d39a81004086bb7ca83578f8"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.403798 25465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:19.423749 25465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:19.426025 25465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:19.426184 25465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:19.426667 25465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:19.426730 25465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.426772 25465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:19.426824 25465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:19.433847 25465 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:32955
I20260808 22:50:19.433957 25656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:32955 every 8 connection(s)
I20260808 22:50:19.434237 25465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:19.435912 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25465
I20260808 22:50:19.436013 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.427123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:48559 (local address 127.22.30.62:40589)
0808 22:50:19.427204 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.427207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.427381 (+ 174us) server_negotiation.cc:407] Connection header received
0808 22:50:19.427831 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.427833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.427846 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.427924 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.428397 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.429155 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.430603 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.430927 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.431004 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.433281 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.433283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.433283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.433290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.434859 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.435177 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.435177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.435178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.435222 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.435446 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.435446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.435447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.435545 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.435610 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.435864 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.435928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:50:19.436035 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:19.436206 22648 external_mini_cluster.cc:1428] Setting key 581ab741660cb083177fc6ecde5d0e66
I20260808 22:50:19.436249 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.427094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.22.30.2:48559)
0808 22:50:19.427242 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.427276 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.427728 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.427968 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.427971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.428173 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.428328 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.428331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.429257 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.429258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.430509 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.430511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.431027 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.431028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.431146 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.431671 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.431680 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.433207 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.434924 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.434928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.434929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.435116 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.435258 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.435259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.435259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.435350 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.435603 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.435607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.435705 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.435828 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.436025 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260808 22:50:19.436796 25658 heartbeater.cc:347] Connected to a master server at 127.22.30.62:40589
I20260808 22:50:19.436882 25658 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:19.439038 25658 heartbeater.cc:511] Master 127.22.30.62:40589 requested a full tablet report, sending...
I20260808 22:50:19.439464 25147 ts_manager.cc:194] Registered new tserver with Master: 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955)
I20260808 22:50:19.439841 25147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:48559
I20260808 22:50:19.444092 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.437730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.22.30.2:56203)
0808 22:50:19.437967 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.437975 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.438022 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.438130 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.438131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.438265 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.438312 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.438313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.439187 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.439188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.440114 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.440116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.440579 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.440580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.440636 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.441028 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.441032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.441288 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.442990 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.442991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.442991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.443149 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.443352 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.443352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.443353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.443457 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.443817 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.443817 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.443851 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.443900 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.444007 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":197,"threads_started":1}
I20260808 22:50:19.444120 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.437796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:56203 (local address 127.22.30.62:40589)
0808 22:50:19.437827 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.437828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.438019 (+ 191us) server_negotiation.cc:407] Connection header received
0808 22:50:19.438059 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.438060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.438070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.438128 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.438374 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.439075 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.440198 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.440496 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.440567 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.441376 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.441377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.441377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.441384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.442903 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.443220 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.443221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.443221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.443281 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.443525 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.443526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.443527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.443735 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.443829 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.443926 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.444001 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:19.462080 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40589
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:19.546196 25676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:19.546402 25676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:19.546444 25676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:19.548372 25676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:19.548447 25676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:19.548481 25676 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 22:50:19.548508 25676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:19.548538 25676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:19.550388 25676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:40589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:19.552237 25676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:19.552546 25676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:19.558385 25683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.558403 25688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:19.558476 25684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:19.558663 25676 server_base.cc:1034] running on GCE node
I20260808 22:50:19.558882 25676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:19.559137 25676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:19.560304 25676 hybrid_clock.cc:648] HybridClock initialized: now 1786229419560294 us; error 27 us; skew 500 ppm
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:19.562645 25676 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:19.563053 25676 webserver.cc:460] Webserver started at http://127.22.30.3:40399/ using document root <none> and password file <none>
I20260808 22:50:19.563272 25676 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:19.563344 25676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:19.563498 25676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:19.564540 25676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "2509d91f96354a608bb00335e2ecdebf"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "b28b76d462a3e9e6c1e99259376c6715"
server_key_iv: "c0c38cb24f31823ef7b2cc79c0c95754"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.564909 25676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "2509d91f96354a608bb00335e2ecdebf"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "b28b76d462a3e9e6c1e99259376c6715"
server_key_iv: "c0c38cb24f31823ef7b2cc79c0c95754"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.566221 25676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:19.566901 25695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.567068 25676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:19.567165 25676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "2509d91f96354a608bb00335e2ecdebf"
format_stamp: "Formatted at 2026-08-08 22:50:19 on dist-test-slave-x4qh"
server_key: "b28b76d462a3e9e6c1e99259376c6715"
server_key_iv: "c0c38cb24f31823ef7b2cc79c0c95754"
server_key_version: "encryptionkey@0"
I20260808 22:50:19.567247 25676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:19.590278 25676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:19.592110 25676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:19.592267 25676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:19.592728 25676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:19.592795 25676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.592832 25676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:19.592862 25676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:19.600646 25676 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:36655
I20260808 22:50:19.600970 25859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:36655 every 8 connection(s)
I20260808 22:50:19.601007 25676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:19.602977 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.593361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:35897 (local address 127.22.30.62:40589)
0808 22:50:19.593426 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.593428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.593641 (+ 213us) server_negotiation.cc:407] Connection header received
0808 22:50:19.594146 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.594148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.594160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.594223 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.595027 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.595887 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.596990 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.597359 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.597440 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.600413 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.600414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.600415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.600422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.601931 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.602268 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.602268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.602269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.602326 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.602494 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.602495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.602495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.602591 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.602637 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.602859 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.602918 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:50:19.603109 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.593369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.22.30.3:35897)
0808 22:50:19.593537 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.593568 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.594046 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.594287 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.594291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.594630 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.594912 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.594917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.596034 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.596035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.596922 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.596924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.597478 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.597479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.597623 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.598225 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.598261 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.600304 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.601998 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.602002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.602003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.602201 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.602367 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.602368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.602368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.602446 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.602645 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.602650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.602751 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.602829 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.602938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260808 22:50:19.603608 25860 heartbeater.cc:347] Connected to a master server at 127.22.30.62:40589
I20260808 22:50:19.603689 25860 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:19.603858 25860 heartbeater.cc:511] Master 127.22.30.62:40589 requested a full tablet report, sending...
I20260808 22:50:19.604209 25145 ts_manager.cc:194] Registered new tserver with Master: 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:19.604710 25145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:35897
I20260808 22:50:19.606787 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25676
I20260808 22:50:19.606909 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:19.607105 22648 external_mini_cluster.cc:1428] Setting key 98a15cfe4889c3ccebc3b8731d464d3f
I20260808 22:50:19.608075 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:50:19.610672 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.604718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:45783 (local address 127.22.30.62:40589)
0808 22:50:19.604784 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.604786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.604830 (+ 44us) server_negotiation.cc:407] Connection header received
0808 22:50:19.604911 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.604912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.604921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.604968 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.605283 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.605823 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.606764 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.607067 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.607169 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.608170 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.608171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.608171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.608178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.609502 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.609899 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.609900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.609900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.609946 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.610148 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.610149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.610149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.610308 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.610374 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.610519 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.610587 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:50:19.610682 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.604587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.22.30.3:45783)
0808 22:50:19.604776 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.604788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.604856 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.604981 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.604982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.605148 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.605204 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.605205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.605926 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.605927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.606688 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.606691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.607173 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.607174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.607266 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.607708 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.607714 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.608083 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.609584 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.609585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.609585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.609827 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.610006 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.610007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.610007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.610078 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.610375 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.610376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.610411 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.610489 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.610586 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:19.631119 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.624745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.22.30.62:40589)
0808 22:50:19.624800 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.624802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.624813 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:50:19.624831 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.624831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.624839 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.624913 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.625419 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.625844 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.626536 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.626843 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.626896 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.628993 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.628994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.628995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.629000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.630271 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.630558 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.630559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.630559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.630604 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.630750 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.630750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.630750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.630834 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.630899 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.631015 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.631067 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:50:19.632762 25145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:19.634217 25145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:19.640383 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.636495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36655 (local address 127.0.0.1:53266)
0808 22:50:19.636763 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.636773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.636829 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.636985 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.636986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.636988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:19.637043 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.637045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.638241 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.638242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.639192 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.639248 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.640110 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.640111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.640184 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.640197 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.640257 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.640315 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":67,"thread_start_us":73,"threads_started":1}
I20260808 22:50:19.640626 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.636688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.22.30.3:36655)
0808 22:50:19.636844 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.636848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.636871 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:50:19.636903 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.636905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.636917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.636986 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:19.637125 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.638168 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.639330 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.640024 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.640095 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.640145 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.640485 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.640526 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260808 22:50:19.640654 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.636443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.22.30.2:32955)
0808 22:50:19.636659 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.636666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.636697 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:50:19.636789 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.636792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.636808 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.636891 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:19.637066 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.638152 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.639587 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.640291 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.640383 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.640435 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.640528 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.640563 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260808 22:50:19.640899 25889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.636320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.0.0.1:46682)
0808 22:50:19.636463 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.636497 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.636558 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.637175 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.637176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.637178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:19.637236 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.637237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.638223 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.638224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.639587 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.639588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.640279 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.640280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.640344 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.640348 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.640394 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.640814 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":93,"thread_start_us":43,"threads_started":1}
I20260808 22:50:19.641042 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.636321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:32955 (local address 127.0.0.1:41488)
0808 22:50:19.636636 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.636648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.636712 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.636904 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.636908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.636910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:19.636974 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.636976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.638264 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.638264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.639507 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.639510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.640390 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.640392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.640450 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.640454 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.640516 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.640936 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":447,"thread_start_us":66,"threads_started":1}
I20260808 22:50:19.641072 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.636490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.22.30.1:32897)
0808 22:50:19.636945 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.636949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.636961 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:50:19.636997 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.636999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.637016 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.637090 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:19.637309 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.638148 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.639655 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.640193 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.640264 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.640316 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.640610 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.640647 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":67,"threads_started":1}
I20260808 22:50:19.641294 25765 tablet_service.cc:1467] Processing CreateTablet for tablet 66024828befc4b07a022edad37411b32 (DEFAULT_TABLE table=test-table [id=6839c1ca7e354fee954a158920ded4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:19.641666 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66024828befc4b07a022edad37411b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.641667 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 66024828befc4b07a022edad37411b32 (DEFAULT_TABLE table=test-table [id=6839c1ca7e354fee954a158920ded4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:19.641687 25563 tablet_service.cc:1467] Processing CreateTablet for tablet 66024828befc4b07a022edad37411b32 (DEFAULT_TABLE table=test-table [id=6839c1ca7e354fee954a158920ded4c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:19.641997 25563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66024828befc4b07a022edad37411b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.642006 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66024828befc4b07a022edad37411b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.645015 25898 tablet_bootstrap.cc:492] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Bootstrap starting.
I20260808 22:50:19.645546 25904 tablet_bootstrap.cc:492] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Bootstrap starting.
I20260808 22:50:19.645818 25898 tablet_bootstrap.cc:654] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.646183 25904 tablet_bootstrap.cc:654] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.646188 25898 log.cc:824] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:19.646543 25904 log.cc:824] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:19.646629 25900 tablet_bootstrap.cc:492] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Bootstrap starting.
I20260808 22:50:19.647300 25904 tablet_bootstrap.cc:492] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: No bootstrap required, opened a new log
I20260808 22:50:19.647300 25898 tablet_bootstrap.cc:492] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: No bootstrap required, opened a new log
I20260808 22:50:19.647404 25904 ts_tablet_manager.cc:1397] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:19.647405 25898 ts_tablet_manager.cc:1397] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:19.647532 25900 tablet_bootstrap.cc:654] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.647832 25900 log.cc:824] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:19.648485 25900 tablet_bootstrap.cc:492] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: No bootstrap required, opened a new log
I20260808 22:50:19.648557 25900 ts_tablet_manager.cc:1397] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:19.649343 25898 raft_consensus.cc:348] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.649381 25904 raft_consensus.cc:348] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.649511 25898 raft_consensus.cc:374] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.649528 25904 raft_consensus.cc:374] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.649562 25904 raft_consensus.cc:729] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4108386c6c1d44bba84a68d542b52b3f, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.649560 25898 raft_consensus.cc:729] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.649672 25904 consensus_queue.cc:260] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.649695 25898 consensus_queue.cc:260] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.649941 25904 ts_tablet_manager.cc:1428] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:50:19.649941 25898 ts_tablet_manager.cc:1428] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:50:19.649981 25658 heartbeater.cc:503] Master 127.22.30.62:40589 was elected leader, sending a full tablet report...
I20260808 22:50:19.650067 25449 heartbeater.cc:503] Master 127.22.30.62:40589 was elected leader, sending a full tablet report...
I20260808 22:50:19.651047 25900 raft_consensus.cc:348] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.651221 25900 raft_consensus.cc:374] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.651268 25900 raft_consensus.cc:729] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.651422 25900 consensus_queue.cc:260] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.651695 25900 ts_tablet_manager.cc:1428] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260808 22:50:19.651757 25860 heartbeater.cc:503] Master 127.22.30.62:40589 was elected leader, sending a full tablet report...
W20260808 22:50:19.684863 25661 tablet.cc:2369] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:19.738083 25911 raft_consensus.cc:482] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:19.738179 25911 raft_consensus.cc:504] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.738445 25911 leader_election.cc:288] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955), 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:19.741423 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.738788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:38221 (local address 127.22.30.2:32955)
0808 22:50:19.738863 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.738865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.738875 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:19.738905 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.738905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.738918 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.738982 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:19.739198 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.739780 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.740721 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.741068 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.741118 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.741126 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.741316 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.741361 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:50:19.741417 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.738705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:32955 (local address 127.22.30.1:38221)
0808 22:50:19.738786 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.738799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.738852 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.739003 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.739005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.739007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:19.739082 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.739100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.739866 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.739867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.740634 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.740636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.741149 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.741150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.741241 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.741244 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.741298 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.741327 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 22:50:19.741488 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.738949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60279 (local address 127.22.30.3:36655)
0808 22:50:19.739005 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.739007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.739155 (+ 148us) server_negotiation.cc:407] Connection header received
0808 22:50:19.739182 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.739183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.739194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.739248 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:19.739417 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.739894 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.740776 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.741085 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.741168 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.741176 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.741385 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.741419 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:50:19.741595 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.738851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36655 (local address 127.22.30.1:60279)
0808 22:50:19.739053 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.739061 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.739120 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.739260 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.739260 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.739262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:19.739333 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.739334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.739970 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.739970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.740666 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.740687 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.741213 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.741214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.741302 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.741306 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.741361 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.741398 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":122,"threads_started":1}
I20260808 22:50:19.741765 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66024828befc4b07a022edad37411b32" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f" is_pre_election: true
I20260808 22:50:19.741930 25589 raft_consensus.cc:2454] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 0.
I20260808 22:50:19.742069 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66024828befc4b07a022edad37411b32" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2509d91f96354a608bb00335e2ecdebf" is_pre_election: true
I20260808 22:50:19.742163 25280 leader_election.cc:302] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4da4311fb4fe3a0e10139a27d791b, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:19.742218 25780 raft_consensus.cc:2454] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 0.
I20260808 22:50:19.742331 25911 raft_consensus.cc:2781] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:19.742416 25911 raft_consensus.cc:482] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:19.742451 25911 raft_consensus.cc:3037] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.743160 25911 raft_consensus.cc:504] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.743291 25911 leader_election.cc:288] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 election: Requested vote from peers 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955), 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:19.743455 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66024828befc4b07a022edad37411b32" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f"
I20260808 22:50:19.743462 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66024828befc4b07a022edad37411b32" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2509d91f96354a608bb00335e2ecdebf"
I20260808 22:50:19.743531 25780 raft_consensus.cc:3037] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.743551 25589 raft_consensus.cc:3037] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.744154 25780 raft_consensus.cc:2454] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 1.
I20260808 22:50:19.744283 25589 raft_consensus.cc:2454] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 1.
I20260808 22:50:19.744305 25280 leader_election.cc:302] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4da4311fb4fe3a0e10139a27d791b, 2509d91f96354a608bb00335e2ecdebf; no voters:
I20260808 22:50:19.744381 25911 raft_consensus.cc:2781] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:19.744545 25911 raft_consensus.cc:686] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 LEADER]: Becoming Leader. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Running, Role: LEADER
I20260808 22:50:19.744747 25911 consensus_queue.cc:237] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } }
I20260808 22:50:19.745582 25145 catalog_manager.cc:5495] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1). New cstate: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } health_report { overall_health: HEALTHY } } }
Aug 08 22:50:19 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229419, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:19.778430 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.771850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40589 (local address 127.0.0.1:50848)
0808 22:50:19.771999 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.772012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.772063 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.772186 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.772187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.772465 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:19.772504 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.772505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.773067 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.773068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.773582 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.773584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.773919 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.773920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.773972 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.774422 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:19.774426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:19.775693 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:19.777291 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.777298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.777300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.777480 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.777723 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:19.777724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:19.777724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:19.777796 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:19.778041 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:19.778049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:19.778090 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.778174 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.778290 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260808 22:50:19.778470 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.771949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.22.30.62:40589)
0808 22:50:19.772013 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.772015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.772063 (+ 48us) server_negotiation.cc:407] Connection header received
0808 22:50:19.772098 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.772098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.772124 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.772170 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:19.772579 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.772999 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.773672 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.773855 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.773905 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.775812 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:19.775813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:19.775814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:19.775820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:19.777191 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.777568 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.777569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.777570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.777632 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:19.777861 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:19.777862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:19.777863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:19.777968 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:19.778039 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.778196 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.778353 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":46}
I20260808 22:50:19.780303 25145 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:50848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 22:50:19.780490 25145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260808 22:50:19.781093 25450 tablet.cc:2369] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:19.781791 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 9550b5aff79c486f84dc073f19f6a316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc6e53bb75fa423a8c570fc765a87067]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:19.781832 25765 tablet_service.cc:1467] Processing CreateTablet for tablet 9550b5aff79c486f84dc073f19f6a316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc6e53bb75fa423a8c570fc765a87067]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:19.781956 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9550b5aff79c486f84dc073f19f6a316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.781973 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9550b5aff79c486f84dc073f19f6a316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.783075 25898 tablet_bootstrap.cc:492] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: Bootstrap starting.
I20260808 22:50:19.783219 25900 tablet_bootstrap.cc:492] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: Bootstrap starting.
I20260808 22:50:19.783344 25563 tablet_service.cc:1467] Processing CreateTablet for tablet 9550b5aff79c486f84dc073f19f6a316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bc6e53bb75fa423a8c570fc765a87067]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:19.783519 25563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9550b5aff79c486f84dc073f19f6a316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.783829 25900 tablet_bootstrap.cc:654] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.783814 25898 tablet_bootstrap.cc:654] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.784466 25898 tablet_bootstrap.cc:492] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: No bootstrap required, opened a new log
I20260808 22:50:19.784528 25898 ts_tablet_manager.cc:1397] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:19.784679 25898 raft_consensus.cc:348] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.784745 25898 raft_consensus.cc:374] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.784770 25898 raft_consensus.cc:729] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.784808 25898 consensus_queue.cc:260] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.784827 25904 tablet_bootstrap.cc:492] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: Bootstrap starting.
I20260808 22:50:19.784909 25898 ts_tablet_manager.cc:1428] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.785292 25900 tablet_bootstrap.cc:492] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: No bootstrap required, opened a new log
I20260808 22:50:19.785339 25900 ts_tablet_manager.cc:1397] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 22:50:19.785459 25904 tablet_bootstrap.cc:654] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.785478 25900 raft_consensus.cc:348] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.785558 25900 raft_consensus.cc:374] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.785584 25900 raft_consensus.cc:729] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.785630 25900 consensus_queue.cc:260] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.785705 25900 ts_tablet_manager.cc:1428] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.785816 25912 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.785873 25911 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.785923 25912 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:19.785959 25911 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:19.786388 25904 tablet_bootstrap.cc:492] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: No bootstrap required, opened a new log
I20260808 22:50:19.786444 25904 ts_tablet_manager.cc:1397] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:19.786586 25904 raft_consensus.cc:348] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.786635 25904 raft_consensus.cc:374] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.786648 25904 raft_consensus.cc:729] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4108386c6c1d44bba84a68d542b52b3f, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.786731 25904 consensus_queue.cc:260] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.786845 25904 ts_tablet_manager.cc:1428] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.786899 25909 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.787001 25909 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:19.833226 25911 raft_consensus.cc:482] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:19.833299 25911 raft_consensus.cc:504] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.833484 25911 leader_election.cc:288] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955), 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:19.833721 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9550b5aff79c486f84dc073f19f6a316" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f" is_pre_election: true
I20260808 22:50:19.833787 25589 raft_consensus.cc:2454] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 0.
I20260808 22:50:19.833738 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9550b5aff79c486f84dc073f19f6a316" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2509d91f96354a608bb00335e2ecdebf" is_pre_election: true
I20260808 22:50:19.833855 25780 raft_consensus.cc:2454] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 0.
I20260808 22:50:19.833930 25280 leader_election.cc:302] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4da4311fb4fe3a0e10139a27d791b, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:19.834020 25911 raft_consensus.cc:2781] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:19.834061 25911 raft_consensus.cc:482] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:19.834079 25911 raft_consensus.cc:3037] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.834688 25911 raft_consensus.cc:504] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.834826 25911 leader_election.cc:288] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 election: Requested vote from peers 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955), 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:19.834985 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9550b5aff79c486f84dc073f19f6a316" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f"
I20260808 22:50:19.835052 25589 raft_consensus.cc:3037] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.835032 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9550b5aff79c486f84dc073f19f6a316" candidate_uuid: "20a4da4311fb4fe3a0e10139a27d791b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2509d91f96354a608bb00335e2ecdebf"
I20260808 22:50:19.835130 25780 raft_consensus.cc:3037] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:19.835697 25589 raft_consensus.cc:2454] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 1.
I20260808 22:50:19.835847 25780 raft_consensus.cc:2454] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a4da4311fb4fe3a0e10139a27d791b in term 1.
I20260808 22:50:19.835867 25280 leader_election.cc:302] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4da4311fb4fe3a0e10139a27d791b, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:19.835980 25911 raft_consensus.cc:2781] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:19.836072 25911 raft_consensus.cc:686] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 LEADER]: Becoming Leader. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Running, Role: LEADER
I20260808 22:50:19.836127 25911 consensus_queue.cc:237] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.836421 25911 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: TxnStatusTablet state changed. Reason: New leader 20a4da4311fb4fe3a0e10139a27d791b. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.836489 25911 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:19.836721 25145 catalog_manager.cc:5495] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1). New cstate: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:19.837031 25931 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 22:50:19.837172 25931 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 22:50:19.845947 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.843165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.0.0.1:46684)
0808 22:50:19.843240 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.843252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.843308 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.843485 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.843486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.843487 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:19.843529 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.843530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.844489 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.844490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.845062 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.845064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.845496 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.845497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.845541 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.845555 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:19.845820 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:19.845826 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.845853 (+ 27us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.845891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:50:19.845956 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.843252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.22.30.1:32897)
0808 22:50:19.843314 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.843317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.843327 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:19.843391 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.843392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.843412 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.843476 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:19.843575 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.844399 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.845134 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.845414 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.845482 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.845635 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:19.845748 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:19.845837 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.845859 (+ 22us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.845880 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:19.847673 25589 raft_consensus.cc:1261] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Refusing update from remote peer 20a4da4311fb4fe3a0e10139a27d791b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:19.847707 25780 raft_consensus.cc:1261] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Refusing update from remote peer 20a4da4311fb4fe3a0e10139a27d791b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:19.848008 25911 consensus_queue.cc:1035] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:19.848145 25915 consensus_queue.cc:1035] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:19.849392 25909 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: TxnStatusTablet state changed. Reason: New leader 20a4da4311fb4fe3a0e10139a27d791b. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.849490 25909 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:19.849503 25912 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: TxnStatusTablet state changed. Reason: New leader 20a4da4311fb4fe3a0e10139a27d791b. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.849613 25912 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 22:50:19.851588 25861 tablet.cc:2369] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:19.852253 25911 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.852329 25911 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:19.852393 25934 mvcc.cc:205] Tried to move back new op lower bound from 7316395703693893632 to 7316395703649255424. Current Snapshot: MvccSnapshot[applied={T|T < 7316395703693893632}]
I20260808 22:50:19.852422 25912 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.852494 25912 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 2509d91f96354a608bb00335e2ecdebf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:19.852547 25935 mvcc.cc:205] Tried to move back new op lower bound from 7316395703693893632 to 7316395703649255424. Current Snapshot: MvccSnapshot[applied={T|T < 7316395703693893632}]
I20260808 22:50:19.852813 25911 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.852896 25911 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 20a4da4311fb4fe3a0e10139a27d791b: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:19.853366 25909 tablet_replica.cc:439] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20a4da4311fb4fe3a0e10139a27d791b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } } }
I20260808 22:50:19.853439 25909 tablet_replica.cc:442] T 9550b5aff79c486f84dc073f19f6a316 P 4108386c6c1d44bba84a68d542b52b3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:19.857436 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.855072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.22.30.1:32897)
0808 22:50:19.855103 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.855105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.855111 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:19.855138 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.855139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.855146 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.855189 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:19.855329 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.855812 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.856903 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.857064 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.857111 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.857271 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:19.857295 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:19.857331 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.857382 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.857395 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:19.863162 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.859783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:50847 (local address 127.22.30.1:32897)
0808 22:50:19.859840 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:19.859843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:19.859987 (+ 144us) server_negotiation.cc:407] Connection header received
0808 22:50:19.860038 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:19.860038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:19.860045 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:19.860082 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:19.860199 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.860593 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.862547 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.862705 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.862786 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.862957 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:19.862987 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:19.863036 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:19.863106 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 22:50:19.863123 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260808 22:50:19.863270 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:19.859724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.22.30.1:50847)
0808 22:50:19.859904 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:19.859914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:19.859985 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:19.860094 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:19.860095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:19.860096 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:19.860147 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.860148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.860663 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.860664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.862468 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:19.862471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:19.862777 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:19.862778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:19.862851 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:19.862866 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:19.863053 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:19.863055 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:19.863104 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:50:19.863148 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":116,"threads_started":1}
I20260808 22:50:19.865331 25780 raft_consensus.cc:1261] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Refusing update from remote peer 20a4da4311fb4fe3a0e10139a27d791b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:19.865353 25589 raft_consensus.cc:1261] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Refusing update from remote peer 20a4da4311fb4fe3a0e10139a27d791b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:19.865542 25950 consensus_queue.cc:1035] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:19.865636 25915 consensus_queue.cc:1035] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:19.867201 25943 mvcc.cc:205] Tried to move back new op lower bound from 7316395703767212032 to 7316395703275085824. Current Snapshot: MvccSnapshot[applied={T|T < 7316395703767212032}]
I20260808 22:50:19.879707 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:19.879786 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:19.880476 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Sending DeleteTablet for 3 replicas of tablet 66024828befc4b07a022edad37411b32
I20260808 22:50:19.880759 25563 tablet_service.cc:1514] Processing DeleteTablet for tablet 66024828befc4b07a022edad37411b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:19 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260808 22:50:19.880777 25765 tablet_service.cc:1514] Processing DeleteTablet for tablet 66024828befc4b07a022edad37411b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:19 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:53266
I20260808 22:50:19.880985 25964 tablet_replica.cc:330] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: stopping tablet replica
I20260808 22:50:19.881074 25964 raft_consensus.cc:2229] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:19.881153 25964 raft_consensus.cc:2258] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:19.881143 25353 tablet_service.cc:1514] Processing DeleteTablet for tablet 66024828befc4b07a022edad37411b32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:19 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46682
I20260808 22:50:19.881264 25145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:19.881394 25145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:19.881575 25964 ts_tablet_manager.cc:1905] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:19.881795 25965 tablet_replica.cc:330] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: stopping tablet replica
I20260808 22:50:19.881887 25965 raft_consensus.cc:2229] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:19.881929 25965 raft_consensus.cc:2258] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:19.882275 25965 ts_tablet_manager.cc:1905] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:19.882493 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 28019dd375a643a394e7788948783774 (DEFAULT_TABLE table=test-table [id=6b7275a3e4d04f3389b00595a3274a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:19.882587 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28019dd375a643a394e7788948783774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.882597 25563 tablet_service.cc:1467] Processing CreateTablet for tablet 28019dd375a643a394e7788948783774 (DEFAULT_TABLE table=test-table [id=6b7275a3e4d04f3389b00595a3274a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:19.882692 25563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28019dd375a643a394e7788948783774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.883471 25964 ts_tablet_manager.cc:1918] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:19.883574 25964 log.cc:1196] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/66024828befc4b07a022edad37411b32
I20260808 22:50:19.883908 25964 ts_tablet_manager.cc:1939] T 66024828befc4b07a022edad37411b32 P 4108386c6c1d44bba84a68d542b52b3f: Deleting consensus metadata
I20260808 22:50:19.884169 25898 tablet_bootstrap.cc:492] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Bootstrap starting.
I20260808 22:50:19.884289 25904 tablet_bootstrap.cc:492] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Bootstrap starting.
I20260808 22:50:19.884932 25904 tablet_bootstrap.cc:654] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.884930 25898 tablet_bootstrap.cc:654] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.885442 25765 tablet_service.cc:1467] Processing CreateTablet for tablet 28019dd375a643a394e7788948783774 (DEFAULT_TABLE table=test-table [id=6b7275a3e4d04f3389b00595a3274a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:19.885532 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28019dd375a643a394e7788948783774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:19.885526 25123 catalog_manager.cc:4841] TS 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955): tablet 66024828befc4b07a022edad37411b32 (table test-table [id=6839c1ca7e354fee954a158920ded4c7]) successfully deleted
I20260808 22:50:19.885993 25904 tablet_bootstrap.cc:492] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: No bootstrap required, opened a new log
I20260808 22:50:19.886044 25904 ts_tablet_manager.cc:1397] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:19.886221 25904 raft_consensus.cc:348] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.886266 25904 raft_consensus.cc:374] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.886282 25904 raft_consensus.cc:729] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4108386c6c1d44bba84a68d542b52b3f, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.886340 25904 consensus_queue.cc:260] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.886430 25904 ts_tablet_manager.cc:1428] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:19.886783 25900 tablet_bootstrap.cc:492] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Bootstrap starting.
I20260808 22:50:19.887162 25900 tablet_bootstrap.cc:654] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:19.887594 25967 tablet_replica.cc:330] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: stopping tablet replica
I20260808 22:50:19.887718 25967 raft_consensus.cc:2229] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:19.887820 25967 raft_consensus.cc:2258] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:19.888295 25900 tablet_bootstrap.cc:492] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: No bootstrap required, opened a new log
I20260808 22:50:19.888368 25900 ts_tablet_manager.cc:1397] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:19.888425 25965 ts_tablet_manager.cc:1918] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:19.888499 25965 log.cc:1196] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/66024828befc4b07a022edad37411b32
I20260808 22:50:19.888513 25900 raft_consensus.cc:348] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.888582 25900 raft_consensus.cc:374] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.888608 25900 raft_consensus.cc:729] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.888660 25900 consensus_queue.cc:260] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.888737 25965 ts_tablet_manager.cc:1939] T 66024828befc4b07a022edad37411b32 P 2509d91f96354a608bb00335e2ecdebf: Deleting consensus metadata
I20260808 22:50:19.888768 25900 ts_tablet_manager.cc:1428] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:19.889258 25123 catalog_manager.cc:4841] TS 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655): tablet 66024828befc4b07a022edad37411b32 (table test-table [id=6839c1ca7e354fee954a158920ded4c7]) successfully deleted
I20260808 22:50:19.891747 25967 ts_tablet_manager.cc:1905] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:19.891906 25898 tablet_bootstrap.cc:492] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: No bootstrap required, opened a new log
I20260808 22:50:19.891956 25898 ts_tablet_manager.cc:1397] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260808 22:50:19.892083 25898 raft_consensus.cc:348] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.892143 25898 raft_consensus.cc:374] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:19.892161 25898 raft_consensus.cc:729] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Initialized, Role: FOLLOWER
I20260808 22:50:19.892194 25898 consensus_queue.cc:260] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:19.892269 25898 ts_tablet_manager.cc:1428] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:19.893317 25967 ts_tablet_manager.cc:1918] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:19.893388 25967 log.cc:1196] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/66024828befc4b07a022edad37411b32
I20260808 22:50:19.893632 25967 ts_tablet_manager.cc:1939] T 66024828befc4b07a022edad37411b32 P 20a4da4311fb4fe3a0e10139a27d791b: Deleting consensus metadata
I20260808 22:50:19.893999 25122 catalog_manager.cc:4841] TS 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897): tablet 66024828befc4b07a022edad37411b32 (table test-table [id=6839c1ca7e354fee954a158920ded4c7]) successfully deleted
I20260808 22:50:20.027235 25912 raft_consensus.cc:482] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:20.027351 25912 raft_consensus.cc:504] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:20.027717 25912 leader_election.cc:288] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897), 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955)
I20260808 22:50:20.030757 25707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.027890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:32955 (local address 127.22.30.3:38731)
0808 22:50:20.027943 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:20.027953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:20.027994 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:20.028199 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:20.028200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:20.028202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:20.028344 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.028347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.028957 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.028958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.029986 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.029989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.030505 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.030506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.030581 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.030584 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:20.030642 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 22:50:20.030678 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:50:20.030743 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.028008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:38731 (local address 127.22.30.2:32955)
0808 22:50:20.028051 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:20.028053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:20.028071 (+ 18us) server_negotiation.cc:407] Connection header received
0808 22:50:20.028101 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:20.028101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:20.028111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:20.028185 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:20.028410 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.028858 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.030081 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.030427 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.030505 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.030514 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:20.030648 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 22:50:20.030688 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:20.031011 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28019dd375a643a394e7788948783774" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f" is_pre_election: true
I20260808 22:50:20.031087 25589 raft_consensus.cc:2454] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 0.
I20260808 22:50:20.031263 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.028152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:39023 (local address 127.22.30.1:32897)
0808 22:50:20.028210 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:20.028212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:20.028219 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:20.028242 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:20.028242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:20.028253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:20.028338 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:20.028941 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.029338 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.030179 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.030643 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.030714 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.030726 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:20.031045 (+ 319us) server_negotiation.cc:299] Negotiation successful
0808 22:50:20.031198 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 22:50:20.031260 25698 leader_election.cc:302] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2509d91f96354a608bb00335e2ecdebf, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:20.031457 25912 raft_consensus.cc:2781] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:20.031493 25912 raft_consensus.cc:482] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:20.031512 25912 raft_consensus.cc:3037] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:20.031760 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.027890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.22.30.3:39023)
0808 22:50:20.028016 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:20.028024 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:20.028062 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:20.028367 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:20.028368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:20.028370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:20.028836 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.028839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.029418 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.029419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.030053 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.030055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.030762 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.030764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.030941 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.030946 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:20.031356 (+ 410us) client_negotiation.cc:240] Negotiation successful
0808 22:50:20.031607 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":15,"thread_start_us":64,"threads_started":1}
I20260808 22:50:20.032027 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28019dd375a643a394e7788948783774" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a4da4311fb4fe3a0e10139a27d791b" is_pre_election: true
I20260808 22:50:20.032163 25381 raft_consensus.cc:2454] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 0.
I20260808 22:50:20.032474 25912 raft_consensus.cc:504] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:20.032608 25912 leader_election.cc:288] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 election: Requested vote from peers 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897), 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955)
I20260808 22:50:20.032743 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28019dd375a643a394e7788948783774" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a4da4311fb4fe3a0e10139a27d791b"
I20260808 22:50:20.032744 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28019dd375a643a394e7788948783774" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f"
I20260808 22:50:20.032823 25381 raft_consensus.cc:3037] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:20.032841 25589 raft_consensus.cc:3037] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:20.033391 25589 raft_consensus.cc:2454] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 1.
I20260808 22:50:20.033557 25698 leader_election.cc:302] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2509d91f96354a608bb00335e2ecdebf, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:20.033653 25912 raft_consensus.cc:2781] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:20.033667 25381 raft_consensus.cc:2454] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 1.
I20260808 22:50:20.033798 25912 raft_consensus.cc:686] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 1 LEADER]: Becoming Leader. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Running, Role: LEADER
I20260808 22:50:20.033870 25912 consensus_queue.cc:237] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:20.034579 25145 catalog_manager.cc:5495] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2509d91f96354a608bb00335e2ecdebf (127.22.30.3). New cstate: current_term: 1 leader_uuid: "2509d91f96354a608bb00335e2ecdebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } health_report { overall_health: HEALTHY } } }
I20260808 22:50:20.073613 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.071353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.22.30.3:36655)
0808 22:50:20.071409 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:20.071411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:20.071418 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:20.071447 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:20.071447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:20.071469 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:20.071520 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:20.071643 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.072019 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.072918 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.073131 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.073179 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.073322 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:20.073407 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:20.073452 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:20.073526 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 22:50:20.073548 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:50:20.078423 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.075907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.22.30.3:50277)
0808 22:50:20.075993 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:20.076005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:20.076043 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:20.076201 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:20.076201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:20.076202 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:20.076286 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.076288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.077143 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.077144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.077689 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.077690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.078115 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.078115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.078159 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.078171 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:20.078319 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:20.078321 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:20.078352 (+ 31us) client_negotiation.cc:240] Negotiation successful
0808 22:50:20.078381 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:50:20.078469 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.075974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:50277 (local address 127.22.30.1:32897)
0808 22:50:20.076025 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:20.076027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:20.076040 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:50:20.076128 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:20.076129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:20.076137 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:20.076200 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:20.076347 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.077050 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.077767 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.078043 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.078117 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.078239 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:20.078268 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:20.078313 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:20.078356 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 22:50:20.078377 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:50:20.080739 25589 raft_consensus.cc:1261] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Refusing update from remote peer 2509d91f96354a608bb00335e2ecdebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:20.080875 25381 raft_consensus.cc:1261] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Refusing update from remote peer 2509d91f96354a608bb00335e2ecdebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:20.081032 25974 consensus_queue.cc:1035] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:20.081468 25974 consensus_queue.cc:1035] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:20.088547 25896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.086361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:40029 (local address 127.22.30.3:36655)
0808 22:50:20.086409 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:20.086411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:20.086417 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:20.086435 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:20.086436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:20.086444 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:20.086495 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:20.086584 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.086997 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.087816 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.088070 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.088159 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.088334 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:20.088363 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:20.088405 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:20.088473 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 22:50:20.088489 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:50:20.088573 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:20.086278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36655 (local address 127.22.30.1:40029)
0808 22:50:20.086320 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:20.086329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:20.086360 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:20.086502 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:20.086503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:20.086504 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:20.086538 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.086539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.087076 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.087077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.087738 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:20.087740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:20.088162 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:20.088164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:20.088215 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:20.088220 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:20.088417 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:20.088419 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:20.088466 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 22:50:20.088504 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 22:50:20.089015 25935 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 22:50:21.612293 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:21.612401 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:21.613137 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Sending DeleteTablet for 3 replicas of tablet 28019dd375a643a394e7788948783774
I20260808 22:50:21.613417 25765 tablet_service.cc:1514] Processing DeleteTablet for tablet 28019dd375a643a394e7788948783774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:21 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:53266
I20260808 22:50:21.613431 25563 tablet_service.cc:1514] Processing DeleteTablet for tablet 28019dd375a643a394e7788948783774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:21 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260808 22:50:21.613623 26020 tablet_replica.cc:330] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: stopping tablet replica
I20260808 22:50:21.613698 26020 raft_consensus.cc:2229] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:21.613783 26020 raft_consensus.cc:2258] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:21.614006 26020 ts_tablet_manager.cc:1905] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:21.615530 26020 ts_tablet_manager.cc:1918] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:50:21.615621 26020 log.cc:1196] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/28019dd375a643a394e7788948783774
I20260808 22:50:21.615881 26020 ts_tablet_manager.cc:1939] T 28019dd375a643a394e7788948783774 P 2509d91f96354a608bb00335e2ecdebf: Deleting consensus metadata
I20260808 22:50:21.616197 25123 catalog_manager.cc:4841] TS 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655): tablet 28019dd375a643a394e7788948783774 (table test-table [id=6b7275a3e4d04f3389b00595a3274a29]) successfully deleted
I20260808 22:50:21.616820 25351 tablet_service.cc:1514] Processing DeleteTablet for tablet 28019dd375a643a394e7788948783774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:21 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46682
I20260808 22:50:21.617075 26023 tablet_replica.cc:330] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: stopping tablet replica
I20260808 22:50:21.617154 26023 raft_consensus.cc:2229] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:21.617228 26023 raft_consensus.cc:2258] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:21.617901 26023 ts_tablet_manager.cc:1905] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:21.617935 26018 tablet_replica.cc:330] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: stopping tablet replica
I20260808 22:50:21.618036 26018 raft_consensus.cc:2229] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:21.618098 26018 raft_consensus.cc:2258] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:21.618312 26018 ts_tablet_manager.cc:1905] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:21.619603 26023 ts_tablet_manager.cc:1918] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:50:21.619688 26023 log.cc:1196] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/28019dd375a643a394e7788948783774
I20260808 22:50:21.619827 26018 ts_tablet_manager.cc:1918] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:50:21.619905 26018 log.cc:1196] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/28019dd375a643a394e7788948783774
I20260808 22:50:21.619908 26023 ts_tablet_manager.cc:1939] T 28019dd375a643a394e7788948783774 P 20a4da4311fb4fe3a0e10139a27d791b: Deleting consensus metadata
I20260808 22:50:21.620155 26018 ts_tablet_manager.cc:1939] T 28019dd375a643a394e7788948783774 P 4108386c6c1d44bba84a68d542b52b3f: Deleting consensus metadata
I20260808 22:50:21.620168 25122 catalog_manager.cc:4841] TS 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897): tablet 28019dd375a643a394e7788948783774 (table test-table [id=6b7275a3e4d04f3389b00595a3274a29]) successfully deleted
I20260808 22:50:21.620441 25123 catalog_manager.cc:4841] TS 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955): tablet 28019dd375a643a394e7788948783774 (table test-table [id=6b7275a3e4d04f3389b00595a3274a29]) successfully deleted
Aug 08 22:50:21 dist-test-slave-x4qh krb5kdc[25077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:50:21 dist-test-slave-x4qh krb5kdc[25077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229421, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:21.636703 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.630238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.22.30.62:40589)
0808 22:50:21.630371 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.630373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.630411 (+ 38us) server_negotiation.cc:407] Connection header received
0808 22:50:21.630464 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.630465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.630474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.630540 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:21.631040 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.631548 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.632185 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.632366 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.632412 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.634280 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:21.634281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:21.634282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:21.634287 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:21.635792 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:21.636089 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:21.636089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:21.636090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:21.636172 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:21.636327 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:21.636327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:21.636327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:21.636414 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:21.636481 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.636565 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.636642 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260808 22:50:21.638199 25145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:21.638363 25145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:21.639585 25351 tablet_service.cc:1467] Processing CreateTablet for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae (DEFAULT_TABLE table=test-table [id=6e1b8e352dfc48c6bfeedddafeb1f766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:21.639614 25765 tablet_service.cc:1467] Processing CreateTablet for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae (DEFAULT_TABLE table=test-table [id=6e1b8e352dfc48c6bfeedddafeb1f766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:21.639632 25563 tablet_service.cc:1467] Processing CreateTablet for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae (DEFAULT_TABLE table=test-table [id=6e1b8e352dfc48c6bfeedddafeb1f766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:21.639710 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:21.639696 25351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:21.639710 25563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:21.640519 26034 tablet_bootstrap.cc:492] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Bootstrap starting.
I20260808 22:50:21.640865 26034 tablet_bootstrap.cc:654] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:21.640973 26035 tablet_bootstrap.cc:492] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Bootstrap starting.
I20260808 22:50:21.641311 26035 tablet_bootstrap.cc:654] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:21.641512 26034 tablet_bootstrap.cc:492] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: No bootstrap required, opened a new log
I20260808 22:50:21.641552 26034 ts_tablet_manager.cc:1397] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:21.641615 26036 tablet_bootstrap.cc:492] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Bootstrap starting.
I20260808 22:50:21.641722 26034 raft_consensus.cc:348] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.641814 26034 raft_consensus.cc:374] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:21.641837 26034 raft_consensus.cc:729] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Initialized, Role: FOLLOWER
I20260808 22:50:21.641929 26034 consensus_queue.cc:260] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.641978 26036 tablet_bootstrap.cc:654] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:21.642010 26035 tablet_bootstrap.cc:492] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: No bootstrap required, opened a new log
I20260808 22:50:21.642086 26035 ts_tablet_manager.cc:1397] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:21.642165 26034 ts_tablet_manager.cc:1428] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:21.642366 26035 raft_consensus.cc:348] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.642467 26035 raft_consensus.cc:374] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:21.642498 26035 raft_consensus.cc:729] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4108386c6c1d44bba84a68d542b52b3f, State: Initialized, Role: FOLLOWER
I20260808 22:50:21.642534 26035 consensus_queue.cc:260] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.642752 26035 ts_tablet_manager.cc:1428] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:21.643163 26036 tablet_bootstrap.cc:492] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: No bootstrap required, opened a new log
I20260808 22:50:21.643280 26036 ts_tablet_manager.cc:1397] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:21.643440 26036 raft_consensus.cc:348] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.643513 26036 raft_consensus.cc:374] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:21.643536 26036 raft_consensus.cc:729] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Initialized, Role: FOLLOWER
I20260808 22:50:21.643646 26036 consensus_queue.cc:260] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.643770 26036 ts_tablet_manager.cc:1428] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:21.816469 26040 raft_consensus.cc:482] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:21.816562 26040 raft_consensus.cc:504] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.816921 26040 leader_election.cc:288] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897), 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:21.820145 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.817061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.22.30.2:34605)
0808 22:50:21.817281 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:21.817298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:21.817357 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:21.817561 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:21.817562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:21.817564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:21.817647 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.817649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.818299 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.818300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.819252 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.819255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.819944 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.819945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.819995 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.819999 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:21.820043 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 22:50:21.820070 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":144,"threads_started":1}
I20260808 22:50:21.820194 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.817125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:34605 (local address 127.22.30.1:32897)
0808 22:50:21.817413 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.817415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.817424 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:21.817452 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.817453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.817464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.817533 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:21.817720 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.818216 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.819342 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.819875 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.819939 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.819949 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.820069 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.820121 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":227,"threads_started":1}
I20260808 22:50:21.820360 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.817167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57339 (local address 127.22.30.3:36655)
0808 22:50:21.817332 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.817334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.817417 (+ 83us) server_negotiation.cc:407] Connection header received
0808 22:50:21.817472 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.817473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.817483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.817572 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:21.817782 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.818239 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.819341 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.819870 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.819939 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.819949 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.820172 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.820224 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260808 22:50:21.820248 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.817098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36655 (local address 127.22.30.2:57339)
0808 22:50:21.817331 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:21.817342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:21.817411 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:21.817598 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:21.817599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:21.817602 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:21.817697 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.817699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.818358 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.818358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.819247 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.819249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.819992 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.819993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.820086 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.820090 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:21.820144 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:50:21.820179 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":51,"thread_start_us":125,"threads_started":1}
I20260808 22:50:21.820709 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a0f9f9c1e241a9bcdc59b1338ff5ae" candidate_uuid: "4108386c6c1d44bba84a68d542b52b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2509d91f96354a608bb00335e2ecdebf" is_pre_election: true
I20260808 22:50:21.820730 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a0f9f9c1e241a9bcdc59b1338ff5ae" candidate_uuid: "4108386c6c1d44bba84a68d542b52b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a4da4311fb4fe3a0e10139a27d791b" is_pre_election: true
I20260808 22:50:21.820806 25780 raft_consensus.cc:2454] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4108386c6c1d44bba84a68d542b52b3f in term 0.
I20260808 22:50:21.820837 25381 raft_consensus.cc:2454] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4108386c6c1d44bba84a68d542b52b3f in term 0.
I20260808 22:50:21.821025 25491 leader_election.cc:302] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2509d91f96354a608bb00335e2ecdebf, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:21.821159 26040 raft_consensus.cc:2781] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:21.821220 26040 raft_consensus.cc:482] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:21.821250 26040 raft_consensus.cc:3037] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:21.822176 26040 raft_consensus.cc:504] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.822317 26040 leader_election.cc:288] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [CANDIDATE]: Term 1 election: Requested vote from peers 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897), 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655)
I20260808 22:50:21.822448 25780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a0f9f9c1e241a9bcdc59b1338ff5ae" candidate_uuid: "4108386c6c1d44bba84a68d542b52b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2509d91f96354a608bb00335e2ecdebf"
I20260808 22:50:21.822476 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a0f9f9c1e241a9bcdc59b1338ff5ae" candidate_uuid: "4108386c6c1d44bba84a68d542b52b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a4da4311fb4fe3a0e10139a27d791b"
I20260808 22:50:21.822538 25780 raft_consensus.cc:3037] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:21.822548 25381 raft_consensus.cc:3037] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:21.823124 25780 raft_consensus.cc:2454] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4108386c6c1d44bba84a68d542b52b3f in term 1.
I20260808 22:50:21.823295 25381 raft_consensus.cc:2454] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4108386c6c1d44bba84a68d542b52b3f in term 1.
I20260808 22:50:21.823321 25491 leader_election.cc:302] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2509d91f96354a608bb00335e2ecdebf, 4108386c6c1d44bba84a68d542b52b3f; no voters:
I20260808 22:50:21.823419 26040 raft_consensus.cc:2781] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:21.823561 26040 raft_consensus.cc:686] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 1 LEADER]: Becoming Leader. State: Replica: 4108386c6c1d44bba84a68d542b52b3f, State: Running, Role: LEADER
I20260808 22:50:21.823640 26040 consensus_queue.cc:237] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.824383 25145 catalog_manager.cc:5495] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2). New cstate: current_term: 1 leader_uuid: "4108386c6c1d44bba84a68d542b52b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:21.870370 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.867837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.22.30.2:32955)
0808 22:50:21.867977 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.867979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.867987 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:21.868031 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.868031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.868047 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.868114 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:21.868248 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.868764 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.869526 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.869740 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.869805 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.870020 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:21.870111 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:21.870163 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.870204 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.870219 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":98,"threads_started":1}
I20260808 22:50:21.875190 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.872879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:44243 (local address 127.22.30.1:32897)
0808 22:50:21.872913 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.872915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.872952 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:50:21.873009 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.873010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.873019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.873062 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:21.873297 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.873883 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.874645 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.874810 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.874854 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.875027 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:21.875049 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:21.875083 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.875114 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.875146 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:50:21.875236 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.872720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.22.30.2:44243)
0808 22:50:21.872902 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:21.872913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:21.872972 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:21.873080 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:21.873080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:21.873081 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:21.873237 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.873238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.873944 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.873944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.874579 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.874581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.874868 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.874868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.874951 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.874963 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:21.875084 (+ 121us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:21.875085 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:21.875112 (+ 27us) client_negotiation.cc:240] Negotiation successful
0808 22:50:21.875155 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260808 22:50:21.877264 25780 raft_consensus.cc:1261] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Refusing update from remote peer 4108386c6c1d44bba84a68d542b52b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:21.877326 25381 raft_consensus.cc:1261] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Refusing update from remote peer 4108386c6c1d44bba84a68d542b52b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:21.877534 26040 consensus_queue.cc:1035] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:21.877698 26052 consensus_queue.cc:1035] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:21.884671 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.882284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:55853 (local address 127.22.30.2:32955)
0808 22:50:21.882321 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.882323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.882457 (+ 134us) server_negotiation.cc:407] Connection header received
0808 22:50:21.882502 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.882503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.882510 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.882550 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:21.882622 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.883018 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.883920 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.884228 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.884300 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.884464 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:21.884491 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:21.884533 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.884596 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.884613 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:21.884712 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.882228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:32955 (local address 127.22.30.1:55853)
0808 22:50:21.882396 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:21.882422 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:21.882466 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:21.882550 (+ 84us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:21.882551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:21.882559 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:21.882590 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.882591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.883071 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.883071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.883753 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:21.883756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.884302 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.884303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:21.884370 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.884374 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:21.884544 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:21.884546 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:21.884593 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 22:50:21.884635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":76,"threads_started":1}
I20260808 22:50:21.890530 25147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:21.890612 25147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:21.891137 25147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Sending DeleteTablet for 3 replicas of tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae
I20260808 22:50:21.891345 25351 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:21 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46682
I20260808 22:50:21.891438 26023 tablet_replica.cc:330] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: stopping tablet replica
I20260808 22:50:21.891484 26023 raft_consensus.cc:2229] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:21.891515 26023 raft_consensus.cc:2258] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:21.891502 25563 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:21 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260808 22:50:21.891450 25765 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:21 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:53266
I20260808 22:50:21.891574 26018 tablet_replica.cc:330] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: stopping tablet replica
I20260808 22:50:21.891584 26020 tablet_replica.cc:330] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: stopping tablet replica
I20260808 22:50:21.891616 26018 raft_consensus.cc:2229] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:21.891634 26020 raft_consensus.cc:2229] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:21.891664 26020 raft_consensus.cc:2258] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:21.891687 26018 raft_consensus.cc:2258] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:21.891985 25147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:21.892099 25147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:21.892087 26023 ts_tablet_manager.cc:1905] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:21.893275 25563 tablet_service.cc:1467] Processing CreateTablet for tablet e20f6c30892a415297a058b2333df3a2 (DEFAULT_TABLE table=test-table [id=aacbb06aac94492fae8808b12a6e297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:21.893329 25765 tablet_service.cc:1467] Processing CreateTablet for tablet e20f6c30892a415297a058b2333df3a2 (DEFAULT_TABLE table=test-table [id=aacbb06aac94492fae8808b12a6e297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:21.893391 25563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20f6c30892a415297a058b2333df3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:21.893424 25765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20f6c30892a415297a058b2333df3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:21.894011 25351 tablet_service.cc:1467] Processing CreateTablet for tablet e20f6c30892a415297a058b2333df3a2 (DEFAULT_TABLE table=test-table [id=aacbb06aac94492fae8808b12a6e297d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:21.894232 26020 ts_tablet_manager.cc:1905] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:21.894912 26035 tablet_bootstrap.cc:492] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Bootstrap starting.
I20260808 22:50:21.894958 26034 tablet_bootstrap.cc:492] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Bootstrap starting.
I20260808 22:50:21.895577 26034 tablet_bootstrap.cc:654] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:21.895984 26035 tablet_bootstrap.cc:654] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:21.896027 26018 ts_tablet_manager.cc:1905] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:21.896597 26034 tablet_bootstrap.cc:492] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: No bootstrap required, opened a new log
I20260808 22:50:21.896644 26034 ts_tablet_manager.cc:1397] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:21.896775 26034 raft_consensus.cc:348] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.896831 26034 raft_consensus.cc:374] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:21.896843 26034 raft_consensus.cc:729] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Initialized, Role: FOLLOWER
I20260808 22:50:21.896901 26034 consensus_queue.cc:260] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.896988 26034 ts_tablet_manager.cc:1428] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:21.897667 26018 ts_tablet_manager.cc:1918] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:21.897716 26018 log.cc:1196] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/97a0f9f9c1e241a9bcdc59b1338ff5ae
I20260808 22:50:21.897933 26018 ts_tablet_manager.cc:1939] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 4108386c6c1d44bba84a68d542b52b3f: Deleting consensus metadata
I20260808 22:50:21.898250 25123 catalog_manager.cc:4841] TS 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955): tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae (table test-table [id=6e1b8e352dfc48c6bfeedddafeb1f766]) successfully deleted
I20260808 22:50:21.898967 26020 ts_tablet_manager.cc:1918] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:21.899030 26020 log.cc:1196] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/97a0f9f9c1e241a9bcdc59b1338ff5ae
I20260808 22:50:21.899231 26020 ts_tablet_manager.cc:1939] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 2509d91f96354a608bb00335e2ecdebf: Deleting consensus metadata
I20260808 22:50:21.899502 26023 ts_tablet_manager.cc:1918] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:21.899509 25351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e20f6c30892a415297a058b2333df3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:21.899578 26023 log.cc:1196] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/97a0f9f9c1e241a9bcdc59b1338ff5ae
I20260808 22:50:21.899590 25123 catalog_manager.cc:4841] TS 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655): tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae (table test-table [id=6e1b8e352dfc48c6bfeedddafeb1f766]) successfully deleted
I20260808 22:50:21.899827 26023 ts_tablet_manager.cc:1939] T 97a0f9f9c1e241a9bcdc59b1338ff5ae P 20a4da4311fb4fe3a0e10139a27d791b: Deleting consensus metadata
I20260808 22:50:21.900151 25122 catalog_manager.cc:4841] TS 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897): tablet 97a0f9f9c1e241a9bcdc59b1338ff5ae (table test-table [id=6e1b8e352dfc48c6bfeedddafeb1f766]) successfully deleted
I20260808 22:50:21.900503 26035 tablet_bootstrap.cc:492] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: No bootstrap required, opened a new log
I20260808 22:50:21.900543 26035 ts_tablet_manager.cc:1397] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260808 22:50:21.900678 26035 raft_consensus.cc:348] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.900728 26035 raft_consensus.cc:374] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:21.900746 26035 raft_consensus.cc:729] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4108386c6c1d44bba84a68d542b52b3f, State: Initialized, Role: FOLLOWER
I20260808 22:50:21.900796 26035 consensus_queue.cc:260] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.900897 26036 tablet_bootstrap.cc:492] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b: Bootstrap starting.
I20260808 22:50:21.901088 26035 ts_tablet_manager.cc:1428] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:21.901232 26036 tablet_bootstrap.cc:654] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:21.901790 26036 tablet_bootstrap.cc:492] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b: No bootstrap required, opened a new log
I20260808 22:50:21.901840 26036 ts_tablet_manager.cc:1397] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:21.901933 26036 raft_consensus.cc:348] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.901983 26036 raft_consensus.cc:374] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:21.902004 26036 raft_consensus.cc:729] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a4da4311fb4fe3a0e10139a27d791b, State: Initialized, Role: FOLLOWER
I20260808 22:50:21.902051 26036 consensus_queue.cc:260] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.902132 26036 ts_tablet_manager.cc:1428] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:21.940819 25987 raft_consensus.cc:482] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:21.940888 25987 raft_consensus.cc:504] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.941021 25987 leader_election.cc:288] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897), 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955)
I20260808 22:50:21.941203 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20f6c30892a415297a058b2333df3a2" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a4da4311fb4fe3a0e10139a27d791b" is_pre_election: true
I20260808 22:50:21.941304 25381 raft_consensus.cc:2454] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 0.
I20260808 22:50:21.941334 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20f6c30892a415297a058b2333df3a2" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f" is_pre_election: true
I20260808 22:50:21.941434 25589 raft_consensus.cc:2454] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 0.
I20260808 22:50:21.941465 25697 leader_election.cc:302] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4da4311fb4fe3a0e10139a27d791b, 2509d91f96354a608bb00335e2ecdebf; no voters:
I20260808 22:50:21.941565 25987 raft_consensus.cc:2781] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:21.941602 25987 raft_consensus.cc:482] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:21.941622 25987 raft_consensus.cc:3037] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:21.942258 25987 raft_consensus.cc:504] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.942384 25987 leader_election.cc:288] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 election: Requested vote from peers 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897), 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955)
I20260808 22:50:21.942503 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20f6c30892a415297a058b2333df3a2" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a4da4311fb4fe3a0e10139a27d791b"
I20260808 22:50:21.942567 25381 raft_consensus.cc:3037] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:21.942570 25589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e20f6c30892a415297a058b2333df3a2" candidate_uuid: "2509d91f96354a608bb00335e2ecdebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4108386c6c1d44bba84a68d542b52b3f"
I20260808 22:50:21.942641 25589 raft_consensus.cc:3037] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:21.943187 25381 raft_consensus.cc:2454] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 1.
I20260808 22:50:21.943351 25697 leader_election.cc:302] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4da4311fb4fe3a0e10139a27d791b, 2509d91f96354a608bb00335e2ecdebf; no voters:
I20260808 22:50:21.943499 25987 raft_consensus.cc:2781] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:21.943548 25987 raft_consensus.cc:686] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 1 LEADER]: Becoming Leader. State: Replica: 2509d91f96354a608bb00335e2ecdebf, State: Running, Role: LEADER
I20260808 22:50:21.943593 25987 consensus_queue.cc:237] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } }
I20260808 22:50:21.943878 25589 raft_consensus.cc:2454] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2509d91f96354a608bb00335e2ecdebf in term 1.
I20260808 22:50:21.944115 25145 catalog_manager.cc:5495] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2509d91f96354a608bb00335e2ecdebf (127.22.30.3). New cstate: current_term: 1 leader_uuid: "2509d91f96354a608bb00335e2ecdebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2509d91f96354a608bb00335e2ecdebf" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36655 } health_report { overall_health: HEALTHY } } }
I20260808 22:50:21.961078 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:21.958677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.22.30.3:36655)
0808 22:50:21.958723 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:21.958725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:21.958732 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:21.958754 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:21.958755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:21.958765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:21.958822 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:21.958928 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.959359 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.960375 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:21.960553 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:21.960604 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:21.960847 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:21.960873 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:21.960911 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:21.961010 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 22:50:21.961028 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 22:50:21.962936 25589 raft_consensus.cc:1261] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Refusing update from remote peer 2509d91f96354a608bb00335e2ecdebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:21.962944 25381 raft_consensus.cc:1261] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b [term 1 FOLLOWER]: Refusing update from remote peer 2509d91f96354a608bb00335e2ecdebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:21.963160 26007 consensus_queue.cc:1035] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a4da4311fb4fe3a0e10139a27d791b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:21.963351 26007 consensus_queue.cc:1035] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4108386c6c1d44bba84a68d542b52b3f" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 22:50:21.968237 25993 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 22:50:23.485948 25145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:23.486055 25145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:23.486907 25145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93025b48668e4797bd2555e2368445bd: Sending DeleteTablet for 3 replicas of tablet e20f6c30892a415297a058b2333df3a2
I20260808 22:50:23.487159 25563 tablet_service.cc:1514] Processing DeleteTablet for tablet e20f6c30892a415297a058b2333df3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:23 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260808 22:50:23.487186 25765 tablet_service.cc:1514] Processing DeleteTablet for tablet e20f6c30892a415297a058b2333df3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:23 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:53266
I20260808 22:50:23.487203 25351 tablet_service.cc:1514] Processing DeleteTablet for tablet e20f6c30892a415297a058b2333df3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:23 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46682
I20260808 22:50:23.487401 26121 tablet_replica.cc:330] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: stopping tablet replica
I20260808 22:50:23.487437 26123 tablet_replica.cc:330] T e20f6c30892a415297a058b2333df3a2 P 20a4da4311fb4fe3a0e10139a27d791b: stopping tablet replica
I20260808 22:50:23.487517 26121 raft_consensus.cc:2229] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:23.487569 26122 tablet_replica.cc:330] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: stopping tablet replica
I20260808 22:50:23.487596 26121 raft_consensus.cc:2258] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:23.487605 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25255
I20260808 22:50:23.487637 26122 raft_consensus.cc:2229] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:23.491205 26121 ts_tablet_manager.cc:1905] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:23.487670 26122 raft_consensus.cc:2258] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:23.493654 26122 ts_tablet_manager.cc:1905] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:23.493660 26121 ts_tablet_manager.cc:1918] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:50:23.493724 26121 log.cc:1196] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/e20f6c30892a415297a058b2333df3a2
I20260808 22:50:23.494017 26121 ts_tablet_manager.cc:1939] T e20f6c30892a415297a058b2333df3a2 P 2509d91f96354a608bb00335e2ecdebf: Deleting consensus metadata
I20260808 22:50:23.494583 25123 catalog_manager.cc:4841] TS 2509d91f96354a608bb00335e2ecdebf (127.22.30.3:36655): tablet e20f6c30892a415297a058b2333df3a2 (table test-table [id=aacbb06aac94492fae8808b12a6e297d]) successfully deleted
I20260808 22:50:23.495178 26122 ts_tablet_manager.cc:1918] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 22:50:23.495224 26122 log.cc:1196] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/e20f6c30892a415297a058b2333df3a2
I20260808 22:50:23.495411 26122 ts_tablet_manager.cc:1939] T e20f6c30892a415297a058b2333df3a2 P 4108386c6c1d44bba84a68d542b52b3f: Deleting consensus metadata
I20260808 22:50:23.496827 25123 catalog_manager.cc:4841] TS 4108386c6c1d44bba84a68d542b52b3f (127.22.30.2:32955): tablet e20f6c30892a415297a058b2333df3a2 (table test-table [id=aacbb06aac94492fae8808b12a6e297d]) successfully deleted
W20260808 22:50:23.507189 25122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:32897 (error 108)
I20260808 22:50:23.507508 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25465
I20260808 22:50:23.512288 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:23.511718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:32897 (local address 127.0.0.1:59298)
0808 22:50:23.512185 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:23.512238 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:32897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":83,"threads_started":1}
W20260808 22:50:23.513556 25122 catalog_manager.cc:4584] TS 20a4da4311fb4fe3a0e10139a27d791b (127.22.30.1:32897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e20f6c30892a415297a058b2333df3a2: Network error: Client connection negotiation failed: client connection to 127.22.30.1:32897: connect: Connection refused (error 111)
I20260808 22:50:23.522840 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25676
I20260808 22:50:23.533097 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25096
2026-08-08T22:50:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6013 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:23 dist-test-slave-x4qh krb5kdc[26129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:23 dist-test-slave-x4qh krb5kdc[26129](info): set up 2 sockets
Aug 08 22:50:23 dist-test-slave-x4qh krb5kdc[26129](info): commencing operation
krb5kdc: starting...
W20260808 22:50:24.698951 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:24 dist-test-slave-x4qh krb5kdc[26129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:24.803193 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42629
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:24.877315 26148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:24.877526 26148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:24.877574 26148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:24.878633 26148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:24.878700 26148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:24.878726 26148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:24.878757 26148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:24.878780 26148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:24.880100 26148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:42629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:24.880327 26148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:24.880523 26148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:24.885180 26156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:24.885241 26157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:24.885193 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:24.885226 26148 server_base.cc:1034] running on GCE node
I20260808 22:50:24.885875 26148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:24.886140 26148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:24.887296 26148 hybrid_clock.cc:648] HybridClock initialized: now 1786229424887286 us; error 43 us; skew 500 ppm
Aug 08 22:50:24 dist-test-slave-x4qh krb5kdc[26129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229424, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:24.889473 26148 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:24.889844 26148 webserver.cc:460] Webserver started at http://127.22.30.62:42159/ using document root <none> and password file <none>
I20260808 22:50:24.890041 26148 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:24.890103 26148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:24.890241 26148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:24.891124 26148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "b59f1885d51747c4915b2f2d256cc269"
format_stamp: "Formatted at 2026-08-08 22:50:24 on dist-test-slave-x4qh"
server_key: "29244754587e8ddd5d3b538ace59eb00"
server_key_iv: "93dea741924a22aef9d8a63dd399f6e4"
server_key_version: "encryptionkey@0"
I20260808 22:50:24.891508 26148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "b59f1885d51747c4915b2f2d256cc269"
format_stamp: "Formatted at 2026-08-08 22:50:24 on dist-test-slave-x4qh"
server_key: "29244754587e8ddd5d3b538ace59eb00"
server_key_iv: "93dea741924a22aef9d8a63dd399f6e4"
server_key_version: "encryptionkey@0"
I20260808 22:50:24.892737 26148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:24.893365 26174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:24.893513 26148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:24.893605 26148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "b59f1885d51747c4915b2f2d256cc269"
format_stamp: "Formatted at 2026-08-08 22:50:24 on dist-test-slave-x4qh"
server_key: "29244754587e8ddd5d3b538ace59eb00"
server_key_iv: "93dea741924a22aef9d8a63dd399f6e4"
server_key_version: "encryptionkey@0"
I20260808 22:50:24.893682 26148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:24.906464 26148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:24.908236 26148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:24.908418 26148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:24.911939 26148 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:42629
I20260808 22:50:24.911971 26260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:42629 every 8 connection(s)
I20260808 22:50:24.912266 26148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:24.913069 26263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:24.915138 26263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269: Bootstrap starting.
I20260808 22:50:24.915745 26263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:24.916046 26263 log.cc:824] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:24.916718 26263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269: No bootstrap required, opened a new log
I20260808 22:50:24.917146 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 26148
I20260808 22:50:24.917246 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:24.917411 22648 external_mini_cluster.cc:1428] Setting key 030e6d7e7254a7f7771179a0e473c12a
I20260808 22:50:24.918594 26263 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59f1885d51747c4915b2f2d256cc269" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42629 } }
I20260808 22:50:24.918803 26263 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:24.918849 26263 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59f1885d51747c4915b2f2d256cc269, State: Initialized, Role: FOLLOWER
I20260808 22:50:24.918970 26263 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59f1885d51747c4915b2f2d256cc269" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42629 } }
I20260808 22:50:24.919054 26263 raft_consensus.cc:388] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:24.919091 26263 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:24.919135 26263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:24.920071 26263 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59f1885d51747c4915b2f2d256cc269" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42629 } }
I20260808 22:50:24.920157 26263 leader_election.cc:302] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59f1885d51747c4915b2f2d256cc269; no voters:
I20260808 22:50:24.920352 26263 leader_election.cc:288] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:24.920377 26268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:24.920482 26268 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [term 1 LEADER]: Becoming Leader. State: Replica: b59f1885d51747c4915b2f2d256cc269, State: Running, Role: LEADER
I20260808 22:50:24.920569 26268 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59f1885d51747c4915b2f2d256cc269" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42629 } }
I20260808 22:50:24.920639 26263 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:24.920909 26274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59f1885d51747c4915b2f2d256cc269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59f1885d51747c4915b2f2d256cc269" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42629 } } }
I20260808 22:50:24.920917 26269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59f1885d51747c4915b2f2d256cc269. Latest consensus state: current_term: 1 leader_uuid: "b59f1885d51747c4915b2f2d256cc269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59f1885d51747c4915b2f2d256cc269" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42629 } } }
I20260808 22:50:24.921100 26274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:24.921118 26269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:24.921505 26282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:24.922015 26282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:24.924238 26282 catalog_manager.cc:1347] Generated new cluster ID: a5b072c70db34fb19009307c691a94f5
I20260808 22:50:24.924284 26282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:24 dist-test-slave-x4qh krb5kdc[26129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:24.927436 26282 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:24.928093 26282 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:24.929018 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:24.917903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.22.30.62:42629)
0808 22:50:24.918064 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:24.918071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:24.918092 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:50:24.918562 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:24.918570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:24.918626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:24.918765 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:24.919609 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:24.920636 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:24.921753 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:24.922448 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:24.922815 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:24.925188 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:24.925192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:24.925193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:24.925203 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:24.927129 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:24.927632 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:24.927635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:24.927635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:24.927776 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:24.928013 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:24.928014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:24.928015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:24.928299 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:24.928462 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:24.928563 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 22:50:24.928742 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260808 22:50:24.931629 26282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b59f1885d51747c4915b2f2d256cc269: Generated new TSK 0
I20260808 22:50:24.931793 26282 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:24.958523 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42629
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:25.041487 26302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:25.041671 26302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:25.041693 26302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:25.042795 26302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:25.042852 26302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:25.042877 26302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:25.044265 26302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:25.044485 26302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:25.044677 26302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:25.050169 26309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:25.050169 26311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:25.050166 26308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:25.050472 26302 server_base.cc:1034] running on GCE node
I20260808 22:50:25.050661 26302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:25.050910 26302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:25.052088 26302 hybrid_clock.cc:648] HybridClock initialized: now 1786229425052069 us; error 30 us; skew 500 ppm
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229425, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:25.054948 26302 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:25.055315 26302 webserver.cc:460] Webserver started at http://127.22.30.1:43961/ using document root <none> and password file <none>
I20260808 22:50:25.055545 26302 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:25.055611 26302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:25.055778 26302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:25.056740 26302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "c579430e8fd34bf6927acdf0467c0b6c"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "9db0d30b8ae99ad6bb879f76b7740871"
server_key_iv: "58852706259ca282808ccc23fc3d8906"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.057154 26302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "c579430e8fd34bf6927acdf0467c0b6c"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "9db0d30b8ae99ad6bb879f76b7740871"
server_key_iv: "58852706259ca282808ccc23fc3d8906"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.058516 26302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:25.059217 26322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.059407 26302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:25.059537 26302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "c579430e8fd34bf6927acdf0467c0b6c"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "9db0d30b8ae99ad6bb879f76b7740871"
server_key_iv: "58852706259ca282808ccc23fc3d8906"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.059625 26302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:25.065826 26302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:25.067799 26302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:25.067950 26302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:25.068153 26302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:25.068444 26302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:25.068495 26302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.068532 26302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:25.068555 26302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.073896 26302 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:40711
I20260808 22:50:25.074005 26496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:40711 every 8 connection(s)
I20260808 22:50:25.074270 26302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229425, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:25.082679 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:25.074904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:39449 (local address 127.22.30.62:42629)
0808 22:50:25.074968 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:25.074970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:25.075169 (+ 199us) server_negotiation.cc:407] Connection header received
0808 22:50:25.075644 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:25.075646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:25.075661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:25.075727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:25.076656 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.077160 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.078118 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.078316 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.078400 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:25.080391 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:25.080392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:25.080392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:25.080398 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:25.081516 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:25.081892 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:25.081893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:25.081893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:25.081939 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:25.082150 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:25.082150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:25.082151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:25.082241 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:25.082294 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:25.082556 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 22:50:25.082621 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:50:25.082837 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:25.074902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42629 (local address 127.22.30.1:39449)
0808 22:50:25.075048 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:25.075084 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:25.075534 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:25.075793 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:25.075796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:25.076348 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:25.076539 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:25.076543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.077284 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.077285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:25.078003 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:25.078005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.078413 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.078414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:25.078524 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:25.078976 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:25.078983 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:25.080300 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:25.081600 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:25.081607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:25.081608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:25.081826 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:25.081996 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:25.081996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:25.081997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:25.082096 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:25.082295 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:25.082299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:25.082424 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 22:50:25.082533 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 22:50:25.082661 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260808 22:50:25.083249 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 26302
I20260808 22:50:25.083297 26497 heartbeater.cc:347] Connected to a master server at 127.22.30.62:42629
I20260808 22:50:25.083343 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:25.083403 26497 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:25.083559 22648 external_mini_cluster.cc:1428] Setting key b79af921a0c3b0fc91adb55c9d5e225b
I20260808 22:50:25.083576 26497 heartbeater.cc:511] Master 127.22.30.62:42629 requested a full tablet report, sending...
I20260808 22:50:25.084077 26202 ts_manager.cc:194] Registered new tserver with Master: c579430e8fd34bf6927acdf0467c0b6c (127.22.30.1:40711)
I20260808 22:50:25.084685 26202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:39449
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:25.107962 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42629
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:25.183946 26505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:25.184157 26505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:25.184197 26505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:25.185387 26505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:25.185451 26505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:25.185505 26505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:25.186926 26505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:25.187184 26505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:25.187438 26505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:25.192129 26515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:25.192241 26505 server_base.cc:1034] running on GCE node
W20260808 22:50:25.192202 26518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:25.192142 26513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:25.192577 26505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:25.192830 26505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:25.193984 26505 hybrid_clock.cc:648] HybridClock initialized: now 1786229425193976 us; error 41 us; skew 500 ppm
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229425, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:25.196370 26505 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:25.196828 26505 webserver.cc:460] Webserver started at http://127.22.30.2:46599/ using document root <none> and password file <none>
I20260808 22:50:25.197036 26505 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:25.197077 26505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:25.197216 26505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:25.198168 26505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "79b58545207c49819550dd337b08c947"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "c82230560b34cf45b365cdcd7aa38df7"
server_key_iv: "423f8c91282b31faff7c6907202ae3b0"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.198509 26505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "79b58545207c49819550dd337b08c947"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "c82230560b34cf45b365cdcd7aa38df7"
server_key_iv: "423f8c91282b31faff7c6907202ae3b0"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.199885 26505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:25.200528 26531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.200712 26505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:25.200793 26505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "79b58545207c49819550dd337b08c947"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "c82230560b34cf45b365cdcd7aa38df7"
server_key_iv: "423f8c91282b31faff7c6907202ae3b0"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.200843 26505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:25.206998 26505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:25.208642 26505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:25.208741 26505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:25.208890 26505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:25.209105 26505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:25.209131 26505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.209147 26505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:25.209159 26505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.214639 26505 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:42265
I20260808 22:50:25.214707 26714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:42265 every 8 connection(s)
I20260808 22:50:25.214980 26505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229425, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:25.222198 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 26505
I20260808 22:50:25.222298 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:25.222494 22648 external_mini_cluster.cc:1428] Setting key e2081a7c211ee56f994fe7e75089a7dd
I20260808 22:50:25.223740 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:25.215633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:40289 (local address 127.22.30.62:42629)
0808 22:50:25.215687 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:25.215703 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:25.215869 (+ 166us) server_negotiation.cc:407] Connection header received
0808 22:50:25.216313 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:25.216314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:25.216327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:25.216398 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:25.217267 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.217752 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.218714 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.218898 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.219015 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:25.221047 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:25.221049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:25.221049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:25.221055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:25.222293 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:25.222813 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:25.222814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:25.222814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:25.222871 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:25.223077 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:25.223078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:25.223078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:25.223193 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:25.223271 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:25.223585 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 22:50:25.223658 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:50:25.223901 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:25.215596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42629 (local address 127.22.30.2:40289)
0808 22:50:25.215759 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:25.215799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:25.216206 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:25.216478 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:25.216482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:25.217002 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:25.217165 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:25.217169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.217844 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.217845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:25.218601 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:25.218603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.219026 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.219027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:25.219146 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:25.219649 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:25.219657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:25.220949 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:25.222454 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:25.222460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:25.222462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:25.222726 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:25.222932 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:25.222933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:25.222933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:25.223021 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:25.223276 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:25.223283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:25.223455 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 22:50:25.223562 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 22:50:25.223697 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260808 22:50:25.224260 26715 heartbeater.cc:347] Connected to a master server at 127.22.30.62:42629
I20260808 22:50:25.224373 26715 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:25.224560 26715 heartbeater.cc:511] Master 127.22.30.62:42629 requested a full tablet report, sending...
I20260808 22:50:25.224910 26202 ts_manager.cc:194] Registered new tserver with Master: 79b58545207c49819550dd337b08c947 (127.22.30.2:42265)
I20260808 22:50:25.225260 26202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:40289
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:25.246059 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42629
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:25.325852 26723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:25.326172 26723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:25.326211 26723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:25.327538 26723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:25.327586 26723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:25.327628 26723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:25.328971 26723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:25.329247 26723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:25.329485 26723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:25.334283 26736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:25.334285 26733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:25.334311 26734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:25.334496 26723 server_base.cc:1034] running on GCE node
I20260808 22:50:25.334702 26723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:25.334941 26723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:25.336090 26723 hybrid_clock.cc:648] HybridClock initialized: now 1786229425336068 us; error 36 us; skew 500 ppm
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229425, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:25.338594 26723 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:25.338940 26723 webserver.cc:460] Webserver started at http://127.22.30.3:33203/ using document root <none> and password file <none>
I20260808 22:50:25.339120 26723 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:25.339169 26723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:25.339244 26723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:25.340210 26723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "70d56a0703de40bd9a03060d209fdb9d"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "32bd77389db11fe23e063e65dd6a7212"
server_key_iv: "a03b14d4884473c9399e8913889ba0e7"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.340502 26723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "70d56a0703de40bd9a03060d209fdb9d"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "32bd77389db11fe23e063e65dd6a7212"
server_key_iv: "a03b14d4884473c9399e8913889ba0e7"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.341857 26723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:25.342548 26748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.342793 26723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:25.342865 26723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "70d56a0703de40bd9a03060d209fdb9d"
format_stamp: "Formatted at 2026-08-08 22:50:25 on dist-test-slave-x4qh"
server_key: "32bd77389db11fe23e063e65dd6a7212"
server_key_iv: "a03b14d4884473c9399e8913889ba0e7"
server_key_version: "encryptionkey@0"
I20260808 22:50:25.342916 26723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:25.359030 26723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:25.360960 26723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:25.361073 26723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:25.361234 26723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:25.361495 26723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:25.361523 26723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.361543 26723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:25.361555 26723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:25.367846 26723 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:43987
I20260808 22:50:25.367936 26929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:43987 every 8 connection(s)
I20260808 22:50:25.368214 26723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:25.370764 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 26723
I20260808 22:50:25.370895 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:25.371083 22648 external_mini_cluster.cc:1428] Setting key 18975d12b79b35c8142c144ff7405838
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229425, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:25.376714 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:25.368914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:47479 (local address 127.22.30.62:42629)
0808 22:50:25.368978 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:25.368980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:25.369161 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:50:25.369619 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:25.369620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:25.369629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:25.369690 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:25.370604 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.371216 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.372377 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.372554 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.372608 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:25.374436 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:25.374437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:25.374437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:25.374443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:25.375616 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:25.375951 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:25.375951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:25.375952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:25.375998 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:25.376215 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:25.376215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:25.376215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:25.376317 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:25.376363 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:25.376593 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 22:50:25.376655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:25.376866 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:25.368866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42629 (local address 127.22.30.3:47479)
0808 22:50:25.369037 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:25.369076 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:25.369539 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:25.369761 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:25.369764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:25.370310 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:25.370520 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:25.370525 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.371325 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.371326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:25.372309 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:25.372311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:25.372626 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:25.372627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:25.372722 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:25.373125 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:25.373133 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:25.374362 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:25.375683 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:25.375688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:25.375690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:25.375889 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:25.376050 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:25.376050 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:25.376050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:25.376156 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:25.376367 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:25.376371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:25.376489 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 22:50:25.376566 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 22:50:25.376674 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260808 22:50:25.377362 26930 heartbeater.cc:347] Connected to a master server at 127.22.30.62:42629
I20260808 22:50:25.377467 26930 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:25.377660 26930 heartbeater.cc:511] Master 127.22.30.62:42629 requested a full tablet report, sending...
I20260808 22:50:25.378000 26202 ts_manager.cc:194] Registered new tserver with Master: 70d56a0703de40bd9a03060d209fdb9d (127.22.30.3:43987)
I20260808 22:50:25.378301 26202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:47479
I20260808 22:50:25.382879 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 22:50:25.395099 26267 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:25.394346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.22.30.62:42629)
0808 22:50:25.394413 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:25.394415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:25.394467 (+ 52us) server_negotiation.cc:407] Connection header received
0808 22:50:25.394507 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:25.394508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:25.394516 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:25.394563 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:25.395038 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35492: BlockingRecv error: recv got EOF from 127.0.0.1:35492 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:50:25.396097 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 26302
I20260808 22:50:25.411931 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 26505
I20260808 22:50:25.420346 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 26723
I20260808 22:50:25.429188 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 26148
2026-08-08T22:50:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1897 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26947](info): set up 2 sockets
Aug 08 22:50:25 dist-test-slave-x4qh krb5kdc[26947](info): commencing operation
krb5kdc: starting...
W20260808 22:50:26.621698 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260808 22:50:26.853973 22648 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.232s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260808 22:50:27.071575 22648 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.217s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260808 22:50:27.200886 22648 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.129s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/test-user.keytab.
W20260808 22:50:27.613440 22648 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.412s user 0.000s sys 0.002s
Aug 08 22:50:27 dist-test-slave-x4qh krb5kdc[26947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:27.655017 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42339
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:42339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:27.731546 26973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:27.731760 26973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:27.731796 26973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:27.732956 26973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:27.733021 26973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:27.733047 26973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:27.733074 26973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:27.733121 26973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:27.734560 26973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:42339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:27.734819 26973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:27.735023 26973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:27.739639 26986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:27.739614 26988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:27.739665 26985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:27.739845 26973 server_base.cc:1034] running on GCE node
I20260808 22:50:27.740060 26973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:27.740338 26973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:27.741479 26973 hybrid_clock.cc:648] HybridClock initialized: now 1786229427741469 us; error 32 us; skew 500 ppm
Aug 08 22:50:27 dist-test-slave-x4qh krb5kdc[26947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229427, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:27.743842 26973 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:27.744289 26973 webserver.cc:460] Webserver started at http://127.22.30.62:42431/ using document root <none> and password file <none>
I20260808 22:50:27.744544 26973 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:27.744588 26973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:27.744755 26973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:27.745754 26973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "ebf6b8d8efb64800ab74ee24504c521a"
format_stamp: "Formatted at 2026-08-08 22:50:27 on dist-test-slave-x4qh"
server_key: "06439888f3805a35332d9bee27933611"
server_key_iv: "8b0a50050845a6855a6df5d8560c2098"
server_key_version: "encryptionkey@0"
I20260808 22:50:27.746106 26973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "ebf6b8d8efb64800ab74ee24504c521a"
format_stamp: "Formatted at 2026-08-08 22:50:27 on dist-test-slave-x4qh"
server_key: "06439888f3805a35332d9bee27933611"
server_key_iv: "8b0a50050845a6855a6df5d8560c2098"
server_key_version: "encryptionkey@0"
I20260808 22:50:27.747430 26973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:27.748137 26999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:27.748292 26973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:27.748394 26973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "ebf6b8d8efb64800ab74ee24504c521a"
format_stamp: "Formatted at 2026-08-08 22:50:27 on dist-test-slave-x4qh"
server_key: "06439888f3805a35332d9bee27933611"
server_key_iv: "8b0a50050845a6855a6df5d8560c2098"
server_key_version: "encryptionkey@0"
I20260808 22:50:27.748466 26973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:27.765799 26973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:27.767573 26973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:27.767743 26973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:27.771792 26973 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:42339
I20260808 22:50:27.771868 27077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:42339 every 8 connection(s)
I20260808 22:50:27.772432 26973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:27.773113 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:27.775106 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a: Bootstrap starting.
I20260808 22:50:27.775668 27082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:27.775926 27082 log.cc:824] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:27.776517 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a: No bootstrap required, opened a new log
I20260808 22:50:27.778636 27082 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6b8d8efb64800ab74ee24504c521a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42339 } }
I20260808 22:50:27.778765 27082 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:27.778789 27082 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf6b8d8efb64800ab74ee24504c521a, State: Initialized, Role: FOLLOWER
I20260808 22:50:27.778913 27082 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6b8d8efb64800ab74ee24504c521a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42339 } }
I20260808 22:50:27.778990 27082 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:27.779021 27082 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:27.779054 27082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:27.779668 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 26973
I20260808 22:50:27.779789 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:27.779954 22648 external_mini_cluster.cc:1428] Setting key 2c69b2a2d9aa701f1907b1c40db91c3b
I20260808 22:50:27.780098 27082 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6b8d8efb64800ab74ee24504c521a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42339 } }
I20260808 22:50:27.780215 27082 leader_election.cc:302] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebf6b8d8efb64800ab74ee24504c521a; no voters:
I20260808 22:50:27.780371 27082 leader_election.cc:288] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:27.780412 27087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:27.780568 27087 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [term 1 LEADER]: Becoming Leader. State: Replica: ebf6b8d8efb64800ab74ee24504c521a, State: Running, Role: LEADER
I20260808 22:50:27.780634 27082 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:27.780746 27087 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6b8d8efb64800ab74ee24504c521a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42339 } }
I20260808 22:50:27.781162 27093 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebf6b8d8efb64800ab74ee24504c521a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6b8d8efb64800ab74ee24504c521a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42339 } } }
I20260808 22:50:27.781257 27093 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:27.781174 27094 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebf6b8d8efb64800ab74ee24504c521a. Latest consensus state: current_term: 1 leader_uuid: "ebf6b8d8efb64800ab74ee24504c521a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf6b8d8efb64800ab74ee24504c521a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42339 } } }
I20260808 22:50:27.781318 27094 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:27.781504 27102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:27.782037 27102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:27.783577 27102 catalog_manager.cc:1347] Generated new cluster ID: 3e17f9b37a1e4f658fcc3499f0ce6a83
I20260808 22:50:27.783631 27102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:27 dist-test-slave-x4qh krb5kdc[26947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:27.791107 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:27.780398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.22.30.62:42339)
0808 22:50:27.780566 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:27.780575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:27.780717 (+ 142us) server_negotiation.cc:407] Connection header received
0808 22:50:27.780900 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:27.780908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:27.780967 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:27.781067 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:27.781997 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:27.782947 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:27.784047 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:27.784640 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:27.784752 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:27.787350 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:27.787354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:27.787355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:27.787365 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:27.789580 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:27.789980 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:27.789982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:27.789983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:27.790072 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:27.790287 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:27.790287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:27.790288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:27.790539 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:27.790677 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:27.790810 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 22:50:27.790924 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260808 22:50:27.792187 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:50:27.796317 27102 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:27.796785 27102 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:27.801584 27102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebf6b8d8efb64800ab74ee24504c521a: Generated new TSK 0
I20260808 22:50:27.801746 27102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:50:27.808234 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:27.804901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.22.30.62:42339)
0808 22:50:27.804935 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:27.804937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:27.805051 (+ 114us) server_negotiation.cc:407] Connection header received
0808 22:50:27.805123 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:27.805124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:27.805145 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:27.805205 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:27.805560 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:27.805991 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:27.806800 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:27.806997 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:27.807071 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:27.807774 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:27.807775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:27.807775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:27.807984 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:27.807998 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:27.808004 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:27.808062 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:27.808144 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 22:50:27.808165 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 22:50:27.810469 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 26973
2026-08-08T22:50:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2387 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:27 dist-test-slave-x4qh krb5kdc[27134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:27 dist-test-slave-x4qh krb5kdc[27134](info): set up 2 sockets
Aug 08 22:50:27 dist-test-slave-x4qh krb5kdc[27134](info): commencing operation
krb5kdc: starting...
W20260808 22:50:28.985903 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:29.093786 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33617
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:33617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:29.169099 27160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:29.169302 27160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:29.169338 27160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:29.170372 27160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:29.170431 27160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:29.170455 27160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:29.170485 27160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:29.170511 27160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:29.171890 27160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:33617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:29.172127 27160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:29.172309 27160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:29.176914 27160 server_base.cc:1034] running on GCE node
W20260808 22:50:29.177011 27173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.176944 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.177016 27169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:29.177392 27160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:29.177683 27160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:29.178821 27160 hybrid_clock.cc:648] HybridClock initialized: now 1786229429178810 us; error 29 us; skew 500 ppm
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:29.181102 27160 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:29.181476 27160 webserver.cc:460] Webserver started at http://127.22.30.62:41515/ using document root <none> and password file <none>
I20260808 22:50:29.181670 27160 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:29.181710 27160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:29.181846 27160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:29.182760 27160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "59d0040a67d44136b5721272d2de4c74"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "c2552d3f15e7d30d60b01601cff3d111"
server_key_iv: "e66f239419fa88f218991474dc3d8fb7"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.183095 27160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "59d0040a67d44136b5721272d2de4c74"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "c2552d3f15e7d30d60b01601cff3d111"
server_key_iv: "e66f239419fa88f218991474dc3d8fb7"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.184369 27160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:29.185004 27183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.185179 27160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:29.185245 27160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "59d0040a67d44136b5721272d2de4c74"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "c2552d3f15e7d30d60b01601cff3d111"
server_key_iv: "e66f239419fa88f218991474dc3d8fb7"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.185360 27160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:29.200500 27160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:29.202116 27160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:29.202244 27160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:29.206055 27160 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:33617
I20260808 22:50:29.206117 27266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:33617 every 8 connection(s)
I20260808 22:50:29.206398 27160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:29.207607 27268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:29.207890 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 27160
I20260808 22:50:29.208035 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:29.208204 22648 external_mini_cluster.cc:1428] Setting key e87f07153fcdf9274a9a3c2be5d9fb3b
I20260808 22:50:29.211040 27268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74: Bootstrap starting.
I20260808 22:50:29.211694 27268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:29.211971 27268 log.cc:824] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:29.212687 27268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74: No bootstrap required, opened a new log
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:29.214532 27268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0040a67d44136b5721272d2de4c74" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33617 } }
I20260808 22:50:29.214747 27268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:29.214803 27268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d0040a67d44136b5721272d2de4c74, State: Initialized, Role: FOLLOWER
I20260808 22:50:29.214937 27268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0040a67d44136b5721272d2de4c74" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33617 } }
I20260808 22:50:29.215051 27268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:29.215101 27268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:29.215153 27268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:29.215796 27268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0040a67d44136b5721272d2de4c74" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33617 } }
I20260808 22:50:29.215916 27268 leader_election.cc:302] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d0040a67d44136b5721272d2de4c74; no voters:
I20260808 22:50:29.216077 27268 leader_election.cc:288] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:29.216243 27279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:29.216338 27268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:29.216432 27279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [term 1 LEADER]: Becoming Leader. State: Replica: 59d0040a67d44136b5721272d2de4c74, State: Running, Role: LEADER
I20260808 22:50:29.216538 27279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0040a67d44136b5721272d2de4c74" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33617 } }
I20260808 22:50:29.216954 27278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59d0040a67d44136b5721272d2de4c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0040a67d44136b5721272d2de4c74" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33617 } } }
I20260808 22:50:29.217051 27278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:29.217304 27280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d0040a67d44136b5721272d2de4c74. Latest consensus state: current_term: 1 leader_uuid: "59d0040a67d44136b5721272d2de4c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0040a67d44136b5721272d2de4c74" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33617 } } }
I20260808 22:50:29.217427 27280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:29.217636 27293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:29.218166 27293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:29.219331 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.208739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.22.30.62:33617)
0808 22:50:29.208929 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:29.208937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:29.208970 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:50:29.209435 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:29.209444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:29.209510 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:29.209657 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:29.210207 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.211416 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.212176 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.212633 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.212735 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.215207 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:29.215228 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:29.215229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:29.215241 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:29.217141 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.217562 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.217565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.217566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.217684 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.217895 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.217896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.217897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.218228 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:29.218390 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:29.218500 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 22:50:29.218930 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260808 22:50:29.219503 27293 catalog_manager.cc:1347] Generated new cluster ID: cf53ebb3ab8d4230b82b27e0320d7591
I20260808 22:50:29.219640 27293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:29.250476 27293 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:29.251000 27293 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:29.255641 27293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59d0040a67d44136b5721272d2de4c74: Generated new TSK 0
I20260808 22:50:29.255807 27293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:29.281414 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33617
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:29.358327 27315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:29.358546 27315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:29.358592 27315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:29.359766 27315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:29.359825 27315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:29.359869 27315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:29.361260 27315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:29.361519 27315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:29.361743 27315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:29.366623 27327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.366629 27323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.366644 27324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:29.366884 27315 server_base.cc:1034] running on GCE node
I20260808 22:50:29.367086 27315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:29.367347 27315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:29.368532 27315 hybrid_clock.cc:648] HybridClock initialized: now 1786229429368513 us; error 40 us; skew 500 ppm
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:29.370889 27315 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:29.371304 27315 webserver.cc:460] Webserver started at http://127.22.30.1:32979/ using document root <none> and password file <none>
I20260808 22:50:29.371551 27315 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:29.371621 27315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:29.371765 27315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:29.372745 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "51cee600c4394da5a9cfc275f1fa6714"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "11ce2da45631e9429962e6e890811114"
server_key_iv: "2540666be36b8187e4869430d8be6d45"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.373080 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "51cee600c4394da5a9cfc275f1fa6714"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "11ce2da45631e9429962e6e890811114"
server_key_iv: "2540666be36b8187e4869430d8be6d45"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.374284 27315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:29.374989 27340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.375164 27315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:29.375262 27315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "51cee600c4394da5a9cfc275f1fa6714"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "11ce2da45631e9429962e6e890811114"
server_key_iv: "2540666be36b8187e4869430d8be6d45"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.375345 27315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:29.394667 27315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:29.396415 27315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:29.396574 27315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:29.396785 27315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:29.397132 27315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:29.397195 27315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.397230 27315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:29.397257 27315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.402720 27315 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:39429
I20260808 22:50:29.402792 27533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:39429 every 8 connection(s)
I20260808 22:50:29.403083 27315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:29.406782 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 27315
I20260808 22:50:29.406872 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:29.407063 22648 external_mini_cluster.cc:1428] Setting key 3be4078e7c1bc368b348ccc2baab3b3e
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:29.412452 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.403881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:48271 (local address 127.22.30.62:33617)
0808 22:50:29.403959 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:29.403961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:29.404099 (+ 138us) server_negotiation.cc:407] Connection header received
0808 22:50:29.404566 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:29.404567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:29.404576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:29.404637 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:29.405555 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.406061 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.407135 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.407432 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.407510 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.409562 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:29.409563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:29.409564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:29.409572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:29.411015 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.411527 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.411527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.411528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.411576 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.411840 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.411841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.411841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.411930 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:29.411976 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:29.412309 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 22:50:29.412387 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:50:29.412680 27540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.403825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33617 (local address 127.22.30.1:48271)
0808 22:50:29.403992 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:29.404025 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:29.404489 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:29.404687 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:29.404690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:29.405279 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:29.405457 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:29.405462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.406151 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.406151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:29.407045 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:29.407047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.407518 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.407519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:29.407616 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.408166 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:29.408181 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:29.409474 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:29.411096 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:29.411104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:29.411106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:29.411460 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:29.411640 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:29.411641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:29.411641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:29.411770 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:29.411994 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:29.412001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:29.412161 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 22:50:29.412274 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 22:50:29.412442 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260808 22:50:29.413271 27534 heartbeater.cc:347] Connected to a master server at 127.22.30.62:33617
I20260808 22:50:29.413384 27534 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:29.413549 27534 heartbeater.cc:511] Master 127.22.30.62:33617 requested a full tablet report, sending...
I20260808 22:50:29.414064 27213 ts_manager.cc:194] Registered new tserver with Master: 51cee600c4394da5a9cfc275f1fa6714 (127.22.30.1:39429)
I20260808 22:50:29.414639 27213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:48271
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:29.431490 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33617
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:29.505591 27543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:29.505888 27543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:29.505925 27543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:29.507046 27543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:29.507085 27543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:29.507134 27543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:29.508538 27543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:29.508752 27543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:29.508960 27543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:29.514802 27553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.514876 27549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.514819 27551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:29.515038 27543 server_base.cc:1034] running on GCE node
I20260808 22:50:29.515234 27543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:29.515453 27543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:29.516610 27543 hybrid_clock.cc:648] HybridClock initialized: now 1786229429516599 us; error 25 us; skew 500 ppm
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:29.523780 27543 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:29.524155 27543 webserver.cc:460] Webserver started at http://127.22.30.2:36629/ using document root <none> and password file <none>
I20260808 22:50:29.524398 27543 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:29.524466 27543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:29.524601 27543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:29.525564 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "5042b6b7eec14fe2b63929b9ee122de9"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "aa22b7c3cd95fb9d2a19314fa480228a"
server_key_iv: "f0eafa0f2ad1b5c75f2a6558cadb411e"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.525911 27543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "5042b6b7eec14fe2b63929b9ee122de9"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "aa22b7c3cd95fb9d2a19314fa480228a"
server_key_iv: "f0eafa0f2ad1b5c75f2a6558cadb411e"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.527114 27543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:29.527865 27564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.528039 27543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.528122 27543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "5042b6b7eec14fe2b63929b9ee122de9"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "aa22b7c3cd95fb9d2a19314fa480228a"
server_key_iv: "f0eafa0f2ad1b5c75f2a6558cadb411e"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.528196 27543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:29.541079 27543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:29.542647 27543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:29.542796 27543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:29.542999 27543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:29.543268 27543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:29.543318 27543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.543349 27543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:29.543423 27543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.549074 27543 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:36713
I20260808 22:50:29.549111 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:36713 every 8 connection(s)
I20260808 22:50:29.549382 27543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:29.555737 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 27543
I20260808 22:50:29.555855 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:29.556067 22648 external_mini_cluster.cc:1428] Setting key 80089de9e7bfd1b700331b658eaa08a0
I20260808 22:50:29.558259 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.550095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:55965 (local address 127.22.30.62:33617)
0808 22:50:29.550162 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:29.550164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:29.550366 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:50:29.550832 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:29.550834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:29.550846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:29.550894 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:29.551798 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.552252 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.553209 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.553431 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.553510 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.555332 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:29.555333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:29.555333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:29.555339 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:29.556906 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.557333 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.557334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.557335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.557391 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.557644 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.557645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.557646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.557760 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:29.557835 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:29.558096 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 22:50:29.558171 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:29.558465 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.550040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33617 (local address 127.22.30.2:55965)
0808 22:50:29.550204 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:29.550257 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:29.550735 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:29.550956 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:29.550960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:29.551537 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:29.551702 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:29.551706 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.552344 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.552344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:29.553071 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:29.553074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.553515 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.553516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:29.553636 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.554019 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:29.554027 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:29.555253 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:29.556999 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:29.557004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:29.557006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:29.557222 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:29.557474 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:29.557474 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:29.557475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:29.557575 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:29.557840 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:29.557846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:29.557971 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 22:50:29.558072 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:50:29.558215 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260808 22:50:29.559062 27747 heartbeater.cc:347] Connected to a master server at 127.22.30.62:33617
I20260808 22:50:29.559144 27747 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:29.559296 27747 heartbeater.cc:511] Master 127.22.30.62:33617 requested a full tablet report, sending...
I20260808 22:50:29.559727 27213 ts_manager.cc:194] Registered new tserver with Master: 5042b6b7eec14fe2b63929b9ee122de9 (127.22.30.2:36713)
I20260808 22:50:29.560109 27213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:55965
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:29.582752 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33617
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:29.659211 27756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:29.659456 27756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:29.659483 27756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:29.660540 27756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:29.660575 27756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:29.660600 27756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:29.661847 27756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:29.661995 27756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:29.662140 27756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:29.666754 27768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:29.666808 27756 server_base.cc:1034] running on GCE node
W20260808 22:50:29.666790 27766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:29.666919 27771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:29.667141 27756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:29.667443 27756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:29.668591 27756 hybrid_clock.cc:648] HybridClock initialized: now 1786229429668577 us; error 26 us; skew 500 ppm
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:29.671023 27756 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:29.671418 27756 webserver.cc:460] Webserver started at http://127.22.30.3:44351/ using document root <none> and password file <none>
I20260808 22:50:29.671661 27756 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:29.671705 27756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:29.671845 27756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:29.672813 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "c1ea0eaab50c4b4d98c4546a59e081f8"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "706429e673c631a6ef2136dc430902c3"
server_key_iv: "f546397b78899cf994b4ede941e4e8bb"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.673164 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "c1ea0eaab50c4b4d98c4546a59e081f8"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "706429e673c631a6ef2136dc430902c3"
server_key_iv: "f546397b78899cf994b4ede941e4e8bb"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.674408 27756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:29.675081 27786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.675232 27756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:29.675333 27756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "c1ea0eaab50c4b4d98c4546a59e081f8"
format_stamp: "Formatted at 2026-08-08 22:50:29 on dist-test-slave-x4qh"
server_key: "706429e673c631a6ef2136dc430902c3"
server_key_iv: "f546397b78899cf994b4ede941e4e8bb"
server_key_version: "encryptionkey@0"
I20260808 22:50:29.675436 27756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:29.689713 27756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:29.691226 27756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:29.691322 27756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:29.691540 27756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:29.691826 27756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:29.691857 27756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.691875 27756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:29.691918 27756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:29.697171 27756 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:40791
I20260808 22:50:29.697255 27972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:40791 every 8 connection(s)
I20260808 22:50:29.697518 27756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:29.705387 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.698171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57023 (local address 127.22.30.62:33617)
0808 22:50:29.698231 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:29.698233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:29.698391 (+ 158us) server_negotiation.cc:407] Connection header received
0808 22:50:29.698815 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:29.698816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:29.698827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:29.698877 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:29.699818 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.700301 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.701148 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.701313 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.701356 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.703183 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:29.703184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:29.703185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:29.703190 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:29.704320 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.704669 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.704670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.704670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.704719 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.704922 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.704922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.704922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.705014 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:29.705062 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:29.705266 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 22:50:29.705328 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:50:29.705513 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.698162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33617 (local address 127.22.30.3:57023)
0808 22:50:29.698305 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:29.698333 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:29.698715 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:29.698946 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:29.698949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:29.699567 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:29.699739 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:29.699743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.700377 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.700377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:29.701079 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:29.701080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.701377 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.701378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:29.701462 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.701863 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:29.701870 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:29.703096 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:29.704405 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:29.704411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:29.704412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:29.704607 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:29.704781 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:29.704781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:29.704782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:29.704869 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:29.705066 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:29.705071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:29.705166 (+ 95us) client_negotiation.cc:769] Sending connection context
0808 22:50:29.705240 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:50:29.705345 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260808 22:50:29.705950 27973 heartbeater.cc:347] Connected to a master server at 127.22.30.62:33617
I20260808 22:50:29.706044 27973 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:29.706197 27973 heartbeater.cc:511] Master 127.22.30.62:33617 requested a full tablet report, sending...
I20260808 22:50:29.706573 27213 ts_manager.cc:194] Registered new tserver with Master: c1ea0eaab50c4b4d98c4546a59e081f8 (127.22.30.3:40791)
I20260808 22:50:29.706892 27213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:57023
I20260808 22:50:29.707171 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 27756
I20260808 22:50:29.707295 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:29.707506 22648 external_mini_cluster.cc:1428] Setting key 5a4e03cc59ec1b8cc50b1cf6692328e9
I20260808 22:50:29.708362 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229429, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:29.730558 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:29.724060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.22.30.62:33617)
0808 22:50:29.724134 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:29.724136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:29.724224 (+ 88us) server_negotiation.cc:407] Connection header received
0808 22:50:29.724308 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:29.724309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:29.724334 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:29.724401 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:29.724922 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.725377 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.726036 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:29.726202 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:29.726272 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:29.728316 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:29.728317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:29.728317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:29.728324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:29.729415 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.729760 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.729761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.729761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.729824 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:29.730023 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:29.730023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:29.730024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:29.730156 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:29.730263 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:29.730375 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 22:50:29.730443 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260808 22:50:29.730846 27213 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42222
I20260808 22:50:29.731274 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 27315
I20260808 22:50:29.744158 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 27543
I20260808 22:50:29.754340 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 27756
I20260808 22:50:29.766286 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 27160
2026-08-08T22:50:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1948 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27994](info): set up 2 sockets
Aug 08 22:50:29 dist-test-slave-x4qh krb5kdc[27994](info): commencing operation
krb5kdc: starting...
W20260808 22:50:30.917945 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:30 dist-test-slave-x4qh krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:31.026268 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39631
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:31.103056 28017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:31.103219 28017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:31.103240 28017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:31.104283 28017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:31.104336 28017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:31.104368 28017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:31.104410 28017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:31.104450 28017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:31.105842 28017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:39631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:31.106037 28017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:31.106230 28017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:31.110634 28027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:31.110685 28024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:31.110620 28023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:31.110780 28017 server_base.cc:1034] running on GCE node
I20260808 22:50:31.110983 28017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:31.111261 28017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:31.112404 28017 hybrid_clock.cc:648] HybridClock initialized: now 1786229431112395 us; error 30 us; skew 500 ppm
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:31.114666 28017 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:31.115131 28017 webserver.cc:460] Webserver started at http://127.22.30.62:35459/ using document root <none> and password file <none>
I20260808 22:50:31.115360 28017 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:31.115461 28017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:31.115597 28017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:31.116474 28017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "4c258fb6c1ce4326bed112e6ba02dece"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "835e07c6c961f28937f36db87a9f83f2"
server_key_iv: "f40e97e5a1c7ff297b22214a41727edd"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.116811 28017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "4c258fb6c1ce4326bed112e6ba02dece"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "835e07c6c961f28937f36db87a9f83f2"
server_key_iv: "f40e97e5a1c7ff297b22214a41727edd"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.118124 28017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:31.118831 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.119004 28017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:31.119097 28017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "4c258fb6c1ce4326bed112e6ba02dece"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "835e07c6c961f28937f36db87a9f83f2"
server_key_iv: "f40e97e5a1c7ff297b22214a41727edd"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.119180 28017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:31.141387 28017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:31.143018 28017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:31.143170 28017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:31.146981 28017 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:39631
I20260808 22:50:31.147042 28119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:39631 every 8 connection(s)
I20260808 22:50:31.147518 28017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:31.148142 28120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:31.150161 28120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece: Bootstrap starting.
I20260808 22:50:31.150686 28120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:31.150923 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 28017
I20260808 22:50:31.151023 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:31.151026 28120 log.cc:824] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:31.151214 22648 external_mini_cluster.cc:1428] Setting key a9742dece34bd8a31dd9479250b5a9d8
I20260808 22:50:31.151749 28120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece: No bootstrap required, opened a new log
I20260808 22:50:31.154213 28120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c258fb6c1ce4326bed112e6ba02dece" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39631 } }
I20260808 22:50:31.154384 28120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:31.154418 28120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c258fb6c1ce4326bed112e6ba02dece, State: Initialized, Role: FOLLOWER
I20260808 22:50:31.154526 28120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c258fb6c1ce4326bed112e6ba02dece" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39631 } }
I20260808 22:50:31.154590 28120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:31.154618 28120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:31.154659 28120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:31.155259 28120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c258fb6c1ce4326bed112e6ba02dece" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39631 } }
I20260808 22:50:31.155412 28120 leader_election.cc:302] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c258fb6c1ce4326bed112e6ba02dece; no voters:
I20260808 22:50:31.155565 28120 leader_election.cc:288] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:31.155786 28120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:31.155639 28127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:31.156015 28127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [term 1 LEADER]: Becoming Leader. State: Replica: 4c258fb6c1ce4326bed112e6ba02dece, State: Running, Role: LEADER
I20260808 22:50:31.156108 28127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c258fb6c1ce4326bed112e6ba02dece" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39631 } }
I20260808 22:50:31.156442 28129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c258fb6c1ce4326bed112e6ba02dece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c258fb6c1ce4326bed112e6ba02dece" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39631 } } }
I20260808 22:50:31.156527 28129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:31.156450 28132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c258fb6c1ce4326bed112e6ba02dece. Latest consensus state: current_term: 1 leader_uuid: "4c258fb6c1ce4326bed112e6ba02dece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c258fb6c1ce4326bed112e6ba02dece" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39631 } } }
I20260808 22:50:31.156742 28132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:31.156744 28145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:31.157243 28145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:31.160321 28145 catalog_manager.cc:1347] Generated new cluster ID: c4645dbc20344770a260aa7cab8addaf
I20260808 22:50:31.160368 28145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:31.161860 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.151682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.22.30.62:39631)
0808 22:50:31.151831 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:31.151844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:31.151866 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:50:31.152214 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:31.152219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:31.152255 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:31.152356 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:31.153095 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.153870 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.154921 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.155358 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.155507 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.158326 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:31.158330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:31.158330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:31.158342 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:31.160169 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.160560 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.160562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.160563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.160664 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.160820 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.160821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.160821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.161053 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:31.161172 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:31.161547 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 22:50:31.161653 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":91,"threads_started":1}
I20260808 22:50:31.169492 28145 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:31.169921 28145 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:31.179481 28145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c258fb6c1ce4326bed112e6ba02dece: Generated new TSK 0
I20260808 22:50:31.179626 28145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:31.207490 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39631
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:31.280401 28164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:31.280596 28164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:31.280637 28164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:31.281687 28164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:31.281726 28164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:31.281750 28164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:31.283087 28164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:31.283284 28164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:31.283552 28164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:31.288033 28178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:31.288056 28174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:31.288057 28175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:31.288173 28164 server_base.cc:1034] running on GCE node
I20260808 22:50:31.288472 28164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:31.288700 28164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:31.289851 28164 hybrid_clock.cc:648] HybridClock initialized: now 1786229431289848 us; error 34 us; skew 500 ppm
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:31.292096 28164 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:31.292512 28164 webserver.cc:460] Webserver started at http://127.22.30.1:37511/ using document root <none> and password file <none>
I20260808 22:50:31.292735 28164 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:31.292804 28164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:31.292954 28164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:31.293900 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "1ab52ca888f444f189f5d7fe6150fa1d"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "5adfdc78acdd24d6a51f7f16f91c6daf"
server_key_iv: "f415801a970f24b46465fbd938f89622"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.294229 28164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "1ab52ca888f444f189f5d7fe6150fa1d"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "5adfdc78acdd24d6a51f7f16f91c6daf"
server_key_iv: "f415801a970f24b46465fbd938f89622"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.295500 28164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:31.296206 28190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.296383 28164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:31.296476 28164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "1ab52ca888f444f189f5d7fe6150fa1d"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "5adfdc78acdd24d6a51f7f16f91c6daf"
server_key_iv: "f415801a970f24b46465fbd938f89622"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.296561 28164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:31.314445 28164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:31.316015 28164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:31.316151 28164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:31.316340 28164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:31.316596 28164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:31.316643 28164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.316670 28164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:31.316694 28164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.322254 28164 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:46347
I20260808 22:50:31.322347 28358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:46347 every 8 connection(s)
I20260808 22:50:31.322615 28164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:31.331146 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.323248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:45821 (local address 127.22.30.62:39631)
0808 22:50:31.323309 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:31.323311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:31.323532 (+ 221us) server_negotiation.cc:407] Connection header received
0808 22:50:31.324410 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:31.324412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:31.324424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:31.324485 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:31.325053 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.325553 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.326563 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.326788 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.326863 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.328781 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:31.328782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:31.328782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:31.328789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:31.329905 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.330264 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.330265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.330265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.330325 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.330556 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.330557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.330557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.330665 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:31.330730 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:31.330981 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 22:50:31.331064 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:31.331272 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.323232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:39631 (local address 127.22.30.1:45821)
0808 22:50:31.323388 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:31.323444 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:31.324251 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:31.324549 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:31.324552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:31.324793 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:31.324955 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:31.324958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.325658 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.325659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:31.326466 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:31.326468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.326884 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.326885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:31.327036 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.327433 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:31.327441 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:31.328665 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:31.329985 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:31.329991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:31.329994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:31.330191 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:31.330388 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:31.330389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:31.330389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:31.330484 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:31.330733 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:31.330740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:31.330856 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 22:50:31.330949 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 22:50:31.331084 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260808 22:50:31.331790 28359 heartbeater.cc:347] Connected to a master server at 127.22.30.62:39631
I20260808 22:50:31.331875 28359 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:31.331902 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 28164
I20260808 22:50:31.332010 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:31.332052 28359 heartbeater.cc:511] Master 127.22.30.62:39631 requested a full tablet report, sending...
I20260808 22:50:31.332197 22648 external_mini_cluster.cc:1428] Setting key 70f5f65286f70efc8f35553cd3364785
I20260808 22:50:31.332595 28066 ts_manager.cc:194] Registered new tserver with Master: 1ab52ca888f444f189f5d7fe6150fa1d (127.22.30.1:46347)
I20260808 22:50:31.333253 28066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:45821
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:31.354909 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39631
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:31.429121 28367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:31.429337 28367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:31.429375 28367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:31.430485 28367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:31.430548 28367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:31.430588 28367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:31.431931 28367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:31.432183 28367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:31.432390 28367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:31.437107 28382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:31.437136 28385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:31.437140 28367 server_base.cc:1034] running on GCE node
W20260808 22:50:31.437112 28374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:31.437469 28367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:31.437701 28367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:31.438850 28367 hybrid_clock.cc:648] HybridClock initialized: now 1786229431438853 us; error 37 us; skew 500 ppm
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:31.441114 28367 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:31.441566 28367 webserver.cc:460] Webserver started at http://127.22.30.2:38465/ using document root <none> and password file <none>
I20260808 22:50:31.441787 28367 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:31.441865 28367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:31.442057 28367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:31.443028 28367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "d580e91661f44460833711fcf0c235cc"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "ca603c5c352b3d459e10db3fa59d7e95"
server_key_iv: "79d8e8ff27df23b58a60d101fbf396a5"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.443434 28367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "d580e91661f44460833711fcf0c235cc"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "ca603c5c352b3d459e10db3fa59d7e95"
server_key_iv: "79d8e8ff27df23b58a60d101fbf396a5"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.444686 28367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:31.445343 28394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.445489 28367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:31.445582 28367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "d580e91661f44460833711fcf0c235cc"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "ca603c5c352b3d459e10db3fa59d7e95"
server_key_iv: "79d8e8ff27df23b58a60d101fbf396a5"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.445663 28367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:31.473206 28367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:31.474696 28367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:31.474843 28367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:31.475064 28367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:31.475437 28367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:31.475497 28367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.475536 28367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:31.475564 28367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.481389 28367 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:36135
I20260808 22:50:31.481479 28586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:36135 every 8 connection(s)
I20260808 22:50:31.481739 28367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:31.489286 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 28367
I20260808 22:50:31.489408 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:31.489603 22648 external_mini_cluster.cc:1428] Setting key e04a16761f01176fb43af1158fb754bf
I20260808 22:50:31.490391 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.482396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:50223 (local address 127.22.30.62:39631)
0808 22:50:31.482480 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:31.482483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:31.482633 (+ 150us) server_negotiation.cc:407] Connection header received
0808 22:50:31.483555 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:31.483556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:31.483569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:31.483615 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:31.484196 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.484712 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.485692 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.485902 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.485977 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.487882 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:31.487884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:31.487884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:31.487892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:31.488942 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.489375 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.489376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.489377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.489439 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.489685 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.489686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.489686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.489839 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:31.489905 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:31.490212 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 22:50:31.490303 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:50:31.490612 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.482356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:39631 (local address 127.22.30.2:50223)
0808 22:50:31.482529 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:31.482558 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:31.483444 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:31.483694 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:31.483697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:31.483932 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:31.484100 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:31.484104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.484808 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.484810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:31.485601 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:31.485603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.485984 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.485985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:31.486097 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.486464 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:31.486471 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:31.487788 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:31.489053 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:31.489060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:31.489062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:31.489286 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:31.489513 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:31.489514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:31.489514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:31.489612 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:31.489916 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:31.489924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:31.490062 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 22:50:31.490179 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 22:50:31.490376 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260808 22:50:31.492511 28587 heartbeater.cc:347] Connected to a master server at 127.22.30.62:39631
I20260808 22:50:31.492641 28587 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:31.492846 28587 heartbeater.cc:511] Master 127.22.30.62:39631 requested a full tablet report, sending...
I20260808 22:50:31.493244 28066 ts_manager.cc:194] Registered new tserver with Master: d580e91661f44460833711fcf0c235cc (127.22.30.2:36135)
I20260808 22:50:31.493605 28066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:50223
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:31.514889 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39631
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:31.590381 28594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:31.590590 28594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:31.590633 28594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:31.591708 28594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:31.591768 28594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:31.591828 28594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:31.593235 28594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:31.593482 28594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:31.593706 28594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:31.598297 28604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:31.598297 28603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:31.598421 28594 server_base.cc:1034] running on GCE node
W20260808 22:50:31.598322 28608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:31.598831 28594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:31.599095 28594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:31.600274 28594 hybrid_clock.cc:648] HybridClock initialized: now 1786229431600259 us; error 25 us; skew 500 ppm
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:31.602680 28594 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:31.603010 28594 webserver.cc:460] Webserver started at http://127.22.30.3:39809/ using document root <none> and password file <none>
I20260808 22:50:31.603176 28594 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:31.603214 28594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:31.603291 28594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:31.604183 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "a4e5085f9cd541ea8ef62f8f5ed6b0b0"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "9cb754e64d5b5c43f6f69467b6083dad"
server_key_iv: "01615fbdd30ffbcb840a5cbe3d870091"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.604463 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "a4e5085f9cd541ea8ef62f8f5ed6b0b0"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "9cb754e64d5b5c43f6f69467b6083dad"
server_key_iv: "01615fbdd30ffbcb840a5cbe3d870091"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.605607 28594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:31.606215 28621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.606392 28594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:31.606456 28594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "a4e5085f9cd541ea8ef62f8f5ed6b0b0"
format_stamp: "Formatted at 2026-08-08 22:50:31 on dist-test-slave-x4qh"
server_key: "9cb754e64d5b5c43f6f69467b6083dad"
server_key_iv: "01615fbdd30ffbcb840a5cbe3d870091"
server_key_version: "encryptionkey@0"
I20260808 22:50:31.606537 28594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:31.632619 28594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:31.634307 28594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:31.634457 28594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:31.634668 28594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:31.634963 28594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:31.634996 28594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.635017 28594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:31.635062 28594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:31.640946 28594 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:34933
I20260808 22:50:31.641000 28806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:34933 every 8 connection(s)
I20260808 22:50:31.641281 28594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:31.649411 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 28594
I20260808 22:50:31.649549 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:31.649765 22648 external_mini_cluster.cc:1428] Setting key b69d7ecc67717669dcdcbe4d9c221787
I20260808 22:50:31.650224 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.641960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36025 (local address 127.22.30.62:39631)
0808 22:50:31.642035 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:31.642037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:31.642216 (+ 179us) server_negotiation.cc:407] Connection header received
0808 22:50:31.643100 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:31.643101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:31.643111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:31.643162 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:31.643791 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.644350 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.645219 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.645435 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.645484 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.647528 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:31.647529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:31.647529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:31.647535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:31.648759 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.649117 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.649117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.649118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.649192 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.649508 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.649509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.649509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.649662 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:31.649714 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:31.650054 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 22:50:31.650136 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:50:31.650413 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.641946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:39631 (local address 127.22.30.3:36025)
0808 22:50:31.642108 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:31.642140 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:31.642994 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:31.643225 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:31.643228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:31.643535 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:31.643714 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:31.643719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.644424 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.644424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:31.645153 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:31.645155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.645508 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.645509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:31.645608 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.646025 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:31.646032 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:31.647446 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:31.648836 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:31.648843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:31.648844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:31.649052 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:31.649287 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:31.649288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:31.649289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:31.649431 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:31.649713 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:31.649720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:31.649881 (+ 161us) client_negotiation.cc:769] Sending connection context
0808 22:50:31.650013 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 22:50:31.650163 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260808 22:50:31.650899 28807 heartbeater.cc:347] Connected to a master server at 127.22.30.62:39631
I20260808 22:50:31.650995 28807 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:31.651175 28807 heartbeater.cc:511] Master 127.22.30.62:39631 requested a full tablet report, sending...
I20260808 22:50:31.651582 28066 ts_manager.cc:194] Registered new tserver with Master: a4e5085f9cd541ea8ef62f8f5ed6b0b0 (127.22.30.3:34933)
I20260808 22:50:31.651948 28066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:36025
I20260808 22:50:31.661558 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:31.683789 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.676667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.22.30.1:46347)
0808 22:50:31.676881 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:31.676885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:31.676895 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:31.676921 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:31.676923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:31.676936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:31.677013 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:31.677547 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.678236 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.678865 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.679147 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.679195 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.681362 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:31.681371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:31.681372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:31.681380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:31.682802 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.683120 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.683123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.683123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.683171 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.683305 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.683305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.683305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.683495 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:31.683597 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:31.683657 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 22:50:31.683718 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[27994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:31.691098 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:31.685123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.22.30.62:39631)
0808 22:50:31.685178 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:31.685180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:31.685186 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:31.685206 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:31.685207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:31.685216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:31.685280 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:31.685703 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.686181 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.686823 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:31.687027 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:31.687100 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:31.689022 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:31.689024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:31.689024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:31.689031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:31.690101 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.690407 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.690408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.690408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.690462 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:31.690660 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:31.690661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:31.690661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:31.690804 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:31.690875 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:31.690987 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 22:50:31.691044 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260808 22:50:31.691299 28066 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51668
I20260808 22:50:31.691615 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 28164
I20260808 22:50:31.701469 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 28367
I20260808 22:50:31.717579 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 28594
I20260808 22:50:31.726038 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 28017
2026-08-08T22:50:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1957 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[28827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[28827](info): set up 2 sockets
Aug 08 22:50:31 dist-test-slave-x4qh krb5kdc[28827](info): commencing operation
krb5kdc: starting...
W20260808 22:50:32.928609 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:33.053979 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:40863
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:33.131201 28851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:33.131399 28851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:33.131421 28851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:33.132382 28851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:33.132412 28851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:33.132426 28851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:33.132437 28851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:33.132448 28851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:33.133704 28851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:33.133873 28851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:33.134024 28851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:33.138366 28861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:33.138440 28851 server_base.cc:1034] running on GCE node
W20260808 22:50:33.138401 28860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:33.138375 28866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:33.138756 28851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:33.139043 28851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:33.140203 28851 hybrid_clock.cc:648] HybridClock initialized: now 1786229433140191 us; error 48 us; skew 500 ppm
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:33.142516 28851 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:33.142841 28851 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:33.142915 28851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:33.143060 28851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:33.144088 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "aae257a0db8344bdbf35401e88ed555d"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "7f7e98eee097bdfcad488e2d58bdd9ca"
server_key_iv: "f341af7f010097707ffee557a1b22657"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.144487 28851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "aae257a0db8344bdbf35401e88ed555d"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "7f7e98eee097bdfcad488e2d58bdd9ca"
server_key_iv: "f341af7f010097707ffee557a1b22657"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.145732 28851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:50:33.146397 28875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.146554 28851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:33.146651 28851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "aae257a0db8344bdbf35401e88ed555d"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "7f7e98eee097bdfcad488e2d58bdd9ca"
server_key_iv: "f341af7f010097707ffee557a1b22657"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.146737 28851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:33.159853 28851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:33.161448 28851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:33.161604 28851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:33.165467 28851 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:40863
I20260808 22:50:33.165493 28976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:40863 every 8 connection(s)
I20260808 22:50:33.165757 28851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:33.166743 28977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:33.168350 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 28851
I20260808 22:50:33.168475 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:33.168659 22648 external_mini_cluster.cc:1428] Setting key 5554b2c4cabd97d68762a4077297f3e0
I20260808 22:50:33.169307 28977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d: Bootstrap starting.
I20260808 22:50:33.169896 28977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:33.170253 28977 log.cc:824] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:33.170957 28977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d: No bootstrap required, opened a new log
I20260808 22:50:33.172976 28977 raft_consensus.cc:348] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae257a0db8344bdbf35401e88ed555d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40863 } }
I20260808 22:50:33.173135 28977 raft_consensus.cc:374] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:33.173175 28977 raft_consensus.cc:729] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae257a0db8344bdbf35401e88ed555d, State: Initialized, Role: FOLLOWER
I20260808 22:50:33.173300 28977 consensus_queue.cc:260] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae257a0db8344bdbf35401e88ed555d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40863 } }
I20260808 22:50:33.173382 28977 raft_consensus.cc:388] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:33.173413 28977 raft_consensus.cc:482] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:33.173466 28977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:33.174186 28977 raft_consensus.cc:504] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae257a0db8344bdbf35401e88ed555d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40863 } }
I20260808 22:50:33.174329 28977 leader_election.cc:302] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aae257a0db8344bdbf35401e88ed555d; no voters:
I20260808 22:50:33.174472 28977 leader_election.cc:288] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:33.174526 28985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:33.174664 28985 raft_consensus.cc:686] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [term 1 LEADER]: Becoming Leader. State: Replica: aae257a0db8344bdbf35401e88ed555d, State: Running, Role: LEADER
I20260808 22:50:33.174713 28977 sys_catalog.cc:564] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:33.174767 28985 consensus_queue.cc:237] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae257a0db8344bdbf35401e88ed555d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40863 } }
I20260808 22:50:33.175065 28988 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aae257a0db8344bdbf35401e88ed555d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae257a0db8344bdbf35401e88ed555d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40863 } } }
I20260808 22:50:33.175148 28988 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [sys.catalog]: This master's current role is: LEADER
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:33.175309 28986 sys_catalog.cc:455] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aae257a0db8344bdbf35401e88ed555d. Latest consensus state: current_term: 1 leader_uuid: "aae257a0db8344bdbf35401e88ed555d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aae257a0db8344bdbf35401e88ed555d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 40863 } } }
I20260808 22:50:33.175376 28986 sys_catalog.cc:458] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:33.175520 28997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:33.175972 28997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:33.178049 28997 catalog_manager.cc:1347] Generated new cluster ID: 7e2f1a0418c44961a795d2677a789a38
I20260808 22:50:33.178112 28997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:33.179234 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.169162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.22.30.62:40863)
0808 22:50:33.169330 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.169336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.169359 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:50:33.169792 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.169798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.169839 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.169945 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:33.170711 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.171644 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.172646 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.172997 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.173073 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.175962 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:33.175965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:33.175966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:33.175977 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:33.177695 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.178046 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.178048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.178049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.178181 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.178382 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.178383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.178384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.178606 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:33.178741 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.178862 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.179089 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260808 22:50:33.183110 28997 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:33.183830 28997 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:33.187965 28997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d: Generated new TSK 0
I20260808 22:50:33.188150 28997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:33.219610 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40863
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:33.315972 29022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:33.316141 29022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:33.316162 29022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:33.317207 29022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:33.317250 29022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:33.317276 29022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:33.318753 29022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:33.318954 29022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:33.319180 29022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:33.323824 29030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:33.323829 29028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:33.324041 29022 server_base.cc:1034] running on GCE node
W20260808 22:50:33.323866 29033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:33.324337 29022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:33.324626 29022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:33.325799 29022 hybrid_clock.cc:648] HybridClock initialized: now 1786229433325779 us; error 30 us; skew 500 ppm
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:33.328513 29022 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:33.328836 29022 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:33.328903 29022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:33.329037 29022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:33.330006 29022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "f902c864bbbf4509a7e187b2db04094b"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "bd1aead93cd1921acf0ceb6411eacd72"
server_key_iv: "119efeb20e1699f6747a5c41307d8287"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.330341 29022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "f902c864bbbf4509a7e187b2db04094b"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "bd1aead93cd1921acf0ceb6411eacd72"
server_key_iv: "119efeb20e1699f6747a5c41307d8287"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.331559 29022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:33.332206 29046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.332376 29022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:33.332463 29022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "f902c864bbbf4509a7e187b2db04094b"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "bd1aead93cd1921acf0ceb6411eacd72"
server_key_iv: "119efeb20e1699f6747a5c41307d8287"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.332537 29022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:33.349411 29022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:33.351541 29022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:33.351686 29022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:33.351890 29022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:33.352173 29022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:33.352219 29022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.352245 29022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:33.352288 29022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.357458 29022 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:41173
I20260808 22:50:33.357493 29245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:41173 every 8 connection(s)
I20260808 22:50:33.357736 29022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:33.364751 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 29022
I20260808 22:50:33.364872 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:33.365088 22648 external_mini_cluster.cc:1428] Setting key 9730c0f316fbb830e526c14e3bc0e758
I20260808 22:50:33.366633 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.358504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:56731 (local address 127.22.30.62:40863)
0808 22:50:33.358565 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.358584 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.358764 (+ 180us) server_negotiation.cc:407] Connection header received
0808 22:50:33.359215 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.359216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.359229 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.359290 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:33.360237 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.360721 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.361673 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.361889 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.361996 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.363908 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:33.363909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:33.363909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:33.363916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:33.365310 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.365707 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.365709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.365709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.365773 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.366017 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.366018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.366019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.366137 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:33.366216 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.366475 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.366554 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:33.366806 29250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.358482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40863 (local address 127.22.30.1:56731)
0808 22:50:33.358650 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.358680 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.359115 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.359358 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.359361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.359636 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:33.360136 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.360141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.360820 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.360821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.361581 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.361583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.362007 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.362008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.362118 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.362470 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:33.362477 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:33.363800 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:33.365408 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:33.365415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:33.365417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:33.365628 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:33.365844 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:33.365845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:33.365845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:33.365947 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:33.366207 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:33.366214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:33.366344 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.366453 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.366586 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260808 22:50:33.367403 29247 heartbeater.cc:347] Connected to a master server at 127.22.30.62:40863
I20260808 22:50:33.367511 29247 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:33.367650 29247 heartbeater.cc:511] Master 127.22.30.62:40863 requested a full tablet report, sending...
I20260808 22:50:33.368119 28900 ts_manager.cc:194] Registered new tserver with Master: f902c864bbbf4509a7e187b2db04094b (127.22.30.1:41173)
I20260808 22:50:33.368690 28900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:56731
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:33.387810 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40863
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:33.460965 29254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:33.461130 29254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:33.461158 29254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:33.462190 29254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:33.462226 29254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:33.462280 29254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:33.463815 29254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:33.464044 29254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:33.464267 29254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:33.468832 29265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:33.468807 29267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:33.468812 29264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:33.469110 29254 server_base.cc:1034] running on GCE node
I20260808 22:50:33.469331 29254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:33.469548 29254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:33.470682 29254 hybrid_clock.cc:648] HybridClock initialized: now 1786229433470664 us; error 41 us; skew 500 ppm
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:33.473028 29254 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:33.473337 29254 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:33.473380 29254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:33.473496 29254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:33.474413 29254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "6c8d2e352dc94c5dadd3f3a37af744dc"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "38f21d7a840daf61aa5b8ce878ebfcf6"
server_key_iv: "48cd7ff9dcc45ba7dd42484824068fab"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.474782 29254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "6c8d2e352dc94c5dadd3f3a37af744dc"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "38f21d7a840daf61aa5b8ce878ebfcf6"
server_key_iv: "48cd7ff9dcc45ba7dd42484824068fab"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.476176 29254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:33.476922 29282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.477100 29254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:33.477186 29254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "6c8d2e352dc94c5dadd3f3a37af744dc"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "38f21d7a840daf61aa5b8ce878ebfcf6"
server_key_iv: "48cd7ff9dcc45ba7dd42484824068fab"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.477291 29254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:33.512910 29254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:33.515228 29254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:33.515376 29254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:33.515628 29254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:33.515918 29254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:33.515947 29254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.515997 29254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:33.516019 29254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.522240 29254 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:44829
I20260808 22:50:33.522313 29475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:44829 every 8 connection(s)
I20260808 22:50:33.522548 29254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:33.530633 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.523171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:42303 (local address 127.22.30.62:40863)
0808 22:50:33.523212 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.523214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.523388 (+ 174us) server_negotiation.cc:407] Connection header received
0808 22:50:33.523887 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.523889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.523901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.523948 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:33.524831 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.525323 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.526268 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.526472 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.526546 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.528539 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:33.528540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:33.528540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:33.528546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:33.529567 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.529903 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.529903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.529903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.529949 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.530160 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.530160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.530160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.530252 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:33.530308 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.530519 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.530578 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:33.530797 29479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.523139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40863 (local address 127.22.30.2:42303)
0808 22:50:33.523259 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.523302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.523771 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.524008 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.524012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.524250 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:33.524726 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.524730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.525422 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.525423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.526179 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.526181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.526551 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.526552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.526670 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.527107 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:33.527114 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:33.528456 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:33.529641 (+ 1185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:33.529647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:33.529648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:33.529841 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:33.530002 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:33.530003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:33.530003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:33.530107 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:33.530312 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:33.530316 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:33.530422 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.530493 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.530591 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260808 22:50:33.531279 29477 heartbeater.cc:347] Connected to a master server at 127.22.30.62:40863
I20260808 22:50:33.531348 29477 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:33.531495 29477 heartbeater.cc:511] Master 127.22.30.62:40863 requested a full tablet report, sending...
I20260808 22:50:33.531818 28900 ts_manager.cc:194] Registered new tserver with Master: 6c8d2e352dc94c5dadd3f3a37af744dc (127.22.30.2:44829)
I20260808 22:50:33.532058 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 29254
I20260808 22:50:33.532128 28900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:42303
I20260808 22:50:33.532166 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:33.532377 22648 external_mini_cluster.cc:1428] Setting key 12d83750ae27854b8071a6c252c1d6dc
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:33.555651 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40863
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:33.632295 29482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:33.632457 29482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:33.632478 29482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:33.633450 29482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:33.633486 29482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:33.633531 29482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:33.634907 29482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:40863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:33.635123 29482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:33.635350 29482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:33.640030 29496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:33.640035 29494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:33.640048 29492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:33.640467 29482 server_base.cc:1034] running on GCE node
I20260808 22:50:33.640691 29482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:33.640965 29482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:33.642125 29482 hybrid_clock.cc:648] HybridClock initialized: now 1786229433642115 us; error 24 us; skew 500 ppm
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:33.644688 29482 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:33.645107 29482 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:33.645195 29482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:33.645356 29482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:33.646337 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "d7a56a35f238414d931577d0248a2098"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "d1bc111cbbfb54d9bf9b15a348c7e63f"
server_key_iv: "6361f8eb1e9861c3488138776231a9a5"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.646682 29482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "d7a56a35f238414d931577d0248a2098"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "d1bc111cbbfb54d9bf9b15a348c7e63f"
server_key_iv: "6361f8eb1e9861c3488138776231a9a5"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.648025 29482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:33.648886 29507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.649080 29482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:33.649170 29482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "d7a56a35f238414d931577d0248a2098"
format_stamp: "Formatted at 2026-08-08 22:50:33 on dist-test-slave-x4qh"
server_key: "d1bc111cbbfb54d9bf9b15a348c7e63f"
server_key_iv: "6361f8eb1e9861c3488138776231a9a5"
server_key_version: "encryptionkey@0"
I20260808 22:50:33.649240 29482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:33.665848 29482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:33.667906 29482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:33.668061 29482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:33.668267 29482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:33.668553 29482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:33.668603 29482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.668637 29482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:33.668661 29482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:33.674398 29482 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:42951
I20260808 22:50:33.674434 29697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:42951 every 8 connection(s)
I20260808 22:50:33.674674 29482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:33.679865 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 29482
I20260808 22:50:33.679987 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:33.680159 22648 external_mini_cluster.cc:1428] Setting key fb963b3691d17ef395b13f8962edcc15
Aug 08 22:50:33 dist-test-slave-x4qh krb5kdc[28827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229433, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:33.682792 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.675268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:51007 (local address 127.22.30.62:40863)
0808 22:50:33.675336 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.675338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.675561 (+ 223us) server_negotiation.cc:407] Connection header received
0808 22:50:33.675980 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.675981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.675993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.676070 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:33.676850 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.677315 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.678202 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.678374 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.678432 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.680681 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:33.680682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:33.680682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:33.680689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:33.681835 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.682152 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.682153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.682153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.682196 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.682381 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.682381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.682382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.682467 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:33.682506 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.682675 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.682731 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:50:33.682931 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.675268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:40863 (local address 127.22.30.3:51007)
0808 22:50:33.675449 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.675488 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.675887 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.676122 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.676125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.676348 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:33.676775 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.676780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.677392 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.677393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.678135 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.678136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.678453 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.678453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.678539 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.678985 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:33.678993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:33.680588 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:33.681893 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:33.681899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:33.681901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:33.682093 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:33.682269 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:33.682269 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:33.682270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:33.682340 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:33.682497 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:33.682499 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:33.682583 (+ 84us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.682645 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.682739 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":121,"threads_started":1}
I20260808 22:50:33.683290 29700 heartbeater.cc:347] Connected to a master server at 127.22.30.62:40863
I20260808 22:50:33.683411 29700 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:33.683539 29700 heartbeater.cc:511] Master 127.22.30.62:40863 requested a full tablet report, sending...
I20260808 22:50:33.683916 28900 ts_manager.cc:194] Registered new tserver with Master: d7a56a35f238414d931577d0248a2098 (127.22.30.3:42951)
I20260808 22:50:33.684262 28900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:51007
I20260808 22:50:33.691887 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:50:33.698204 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.693116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.22.30.62:40863)
0808 22:50:33.693187 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.693189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.693292 (+ 103us) server_negotiation.cc:407] Connection header received
0808 22:50:33.693319 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.693320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.693327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.693374 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:33.693842 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.694260 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.694888 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.695040 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.695082 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.696150 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:33.696151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:33.696151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:33.696156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:33.697292 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.697577 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.697578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.697578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.697624 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:33.697753 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:33.697754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:33.697754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:33.697872 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:33.697940 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.698048 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.698104 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260808 22:50:33.699592 28900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:33.701100 28900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:33.707259 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.703206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.22.30.3:42951)
0808 22:50:33.703349 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.703353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.703383 (+ 30us) server_negotiation.cc:407] Connection header received
0808 22:50:33.703418 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.703420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.703435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.703698 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:33.703772 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.704689 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706188 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.706812 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706883 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.706936 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.707125 (+ 189us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.707175 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260808 22:50:33.707260 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.703327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:41173 (local address 127.0.0.1:34850)
0808 22:50:33.703457 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.703465 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.703502 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.704099 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.704100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.704102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:33.704150 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.704152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.705329 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.705330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.706363 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.706365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.706977 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.707024 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.707044 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.707074 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.707144 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":99,"threads_started":1}
I20260808 22:50:33.707473 29723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.703130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:42951 (local address 127.0.0.1:35412)
0808 22:50:33.703224 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.703260 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.703314 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.703552 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.703555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.703557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:33.703623 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.703625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.704812 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.704813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.706102 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.706104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.706904 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.706984 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.706997 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.707085 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.707145 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":22,"thread_start_us":56,"threads_started":1}
I20260808 22:50:33.707479 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.703535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.22.30.1:41173)
0808 22:50:33.703943 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.703948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.703956 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:33.703990 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.703992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.704005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.704225 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:33.704332 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.705193 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706448 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.706893 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.707228 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.707301 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.707328 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.707395 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":85,"threads_started":1}
I20260808 22:50:33.707629 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.703521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:44829 (local address 127.0.0.1:37662)
0808 22:50:33.703608 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.703617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.703653 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.704148 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.704149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.704151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:33.704201 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.704202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.705285 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.705286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.706077 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.706079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.706983 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.707094 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.707099 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.707156 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.707195 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":60,"threads_started":1}
I20260808 22:50:33.707892 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.703721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.22.30.2:44829)
0808 22:50:33.703883 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.703887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.703897 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:33.703936 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.703938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.703954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.704037 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:33.704288 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.705180 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706164 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.706822 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.706883 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.706936 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.707752 (+ 816us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.707800 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260808 22:50:33.708132 29587 tablet_service.cc:1467] Processing CreateTablet for tablet c3518be9b55c4f03a29372b8d519b753 (DEFAULT_TABLE table=test-table [id=c9361464b8ce4bd082612fc408179390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:33.708122 29135 tablet_service.cc:1467] Processing CreateTablet for tablet c3518be9b55c4f03a29372b8d519b753 (DEFAULT_TABLE table=test-table [id=c9361464b8ce4bd082612fc408179390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:33.708483 29587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3518be9b55c4f03a29372b8d519b753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:33.708483 29135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3518be9b55c4f03a29372b8d519b753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:33.708550 29365 tablet_service.cc:1467] Processing CreateTablet for tablet c3518be9b55c4f03a29372b8d519b753 (DEFAULT_TABLE table=test-table [id=c9361464b8ce4bd082612fc408179390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:33.708879 29365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3518be9b55c4f03a29372b8d519b753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:33.711200 29731 tablet_bootstrap.cc:492] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Bootstrap starting.
I20260808 22:50:33.711910 29736 tablet_bootstrap.cc:492] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Bootstrap starting.
I20260808 22:50:33.711967 29731 tablet_bootstrap.cc:654] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:33.712479 29736 tablet_bootstrap.cc:654] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:33.712531 29731 log.cc:824] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:33.712860 29738 tablet_bootstrap.cc:492] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: Bootstrap starting.
I20260808 22:50:33.713099 29736 log.cc:824] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:33.713462 29731 tablet_bootstrap.cc:492] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: No bootstrap required, opened a new log
I20260808 22:50:33.713501 29738 tablet_bootstrap.cc:654] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:33.713577 29731 ts_tablet_manager.cc:1397] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:33.713769 29736 tablet_bootstrap.cc:492] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: No bootstrap required, opened a new log
I20260808 22:50:33.713847 29736 ts_tablet_manager.cc:1397] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:33.714040 29738 log.cc:824] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:33.714816 29738 tablet_bootstrap.cc:492] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: No bootstrap required, opened a new log
I20260808 22:50:33.714913 29738 ts_tablet_manager.cc:1397] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:33.715334 29731 raft_consensus.cc:348] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.715508 29731 raft_consensus.cc:374] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:33.715560 29731 raft_consensus.cc:729] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8d2e352dc94c5dadd3f3a37af744dc, State: Initialized, Role: FOLLOWER
I20260808 22:50:33.715668 29731 consensus_queue.cc:260] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.715741 29736 raft_consensus.cc:348] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.715894 29736 raft_consensus.cc:374] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:33.715936 29736 raft_consensus.cc:729] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a56a35f238414d931577d0248a2098, State: Initialized, Role: FOLLOWER
I20260808 22:50:33.715951 29731 ts_tablet_manager.cc:1428] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:33.715981 29477 heartbeater.cc:503] Master 127.22.30.62:40863 was elected leader, sending a full tablet report...
I20260808 22:50:33.716034 29736 consensus_queue.cc:260] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.716305 29700 heartbeater.cc:503] Master 127.22.30.62:40863 was elected leader, sending a full tablet report...
I20260808 22:50:33.716295 29736 ts_tablet_manager.cc:1428] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 22:50:33.717710 29738 raft_consensus.cc:348] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.717859 29738 raft_consensus.cc:374] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:33.717909 29738 raft_consensus.cc:729] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f902c864bbbf4509a7e187b2db04094b, State: Initialized, Role: FOLLOWER
I20260808 22:50:33.718026 29738 consensus_queue.cc:260] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.718276 29738 ts_tablet_manager.cc:1428] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:50:33.718320 29247 heartbeater.cc:503] Master 127.22.30.62:40863 was elected leader, sending a full tablet report...
W20260808 22:50:33.773151 29478 tablet.cc:2369] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:33.821199 29743 raft_consensus.cc:482] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:33.821362 29743 raft_consensus.cc:504] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.821731 29743 leader_election.cc:288] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7a56a35f238414d931577d0248a2098 (127.22.30.3:42951), f902c864bbbf4509a7e187b2db04094b (127.22.30.1:41173)
I20260808 22:50:33.824597 29479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.821945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:42951 (local address 127.22.30.2:45097)
0808 22:50:33.822014 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.822029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.822084 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.822242 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.822243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.822244 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:33.822329 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.822330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.822953 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.822954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.823763 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.823765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.824302 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.824303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.824421 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.824424 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.824492 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.824528 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 22:50:33.824597 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.822036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:45097 (local address 127.22.30.3:42951)
0808 22:50:33.822085 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.822087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.822096 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:33.822158 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.822159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.822171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.822224 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:33.822405 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.822872 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.823858 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.824220 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.824301 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.824343 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.824498 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.824536 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:50:33.824735 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.822147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:45509 (local address 127.22.30.1:41173)
0808 22:50:33.822199 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.822201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.822333 (+ 132us) server_negotiation.cc:407] Connection header received
0808 22:50:33.822401 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.822403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.822414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.822467 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:33.822628 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.823076 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.824020 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.824392 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.824473 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.824482 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.824638 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.824672 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:50:33.824748 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.822071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:41173 (local address 127.22.30.2:45509)
0808 22:50:33.822248 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.822263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:33.822343 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:33.822479 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:33.822480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:33.822481 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:33.822555 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.822557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.823160 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.823161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.823930 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:33.823932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.824488 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.824489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:33.824562 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.824565 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:33.824630 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:50:33.824669 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":124,"threads_started":1}
I20260808 22:50:33.824956 29619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3518be9b55c4f03a29372b8d519b753" candidate_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7a56a35f238414d931577d0248a2098" is_pre_election: true
I20260808 22:50:33.825106 29619 raft_consensus.cc:2454] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d2e352dc94c5dadd3f3a37af744dc in term 0.
I20260808 22:50:33.825321 29289 leader_election.cc:302] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d2e352dc94c5dadd3f3a37af744dc, d7a56a35f238414d931577d0248a2098; no voters:
I20260808 22:50:33.825486 29743 raft_consensus.cc:2781] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:33.825469 29170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3518be9b55c4f03a29372b8d519b753" candidate_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f902c864bbbf4509a7e187b2db04094b" is_pre_election: true
I20260808 22:50:33.825532 29743 raft_consensus.cc:482] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:33.825546 29743 raft_consensus.cc:3037] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:33.825620 29170 raft_consensus.cc:2454] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c8d2e352dc94c5dadd3f3a37af744dc in term 0.
I20260808 22:50:33.826300 29743 raft_consensus.cc:504] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.826426 29743 leader_election.cc:288] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [CANDIDATE]: Term 1 election: Requested vote from peers d7a56a35f238414d931577d0248a2098 (127.22.30.3:42951), f902c864bbbf4509a7e187b2db04094b (127.22.30.1:41173)
I20260808 22:50:33.826586 29619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3518be9b55c4f03a29372b8d519b753" candidate_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7a56a35f238414d931577d0248a2098"
I20260808 22:50:33.826627 29170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3518be9b55c4f03a29372b8d519b753" candidate_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f902c864bbbf4509a7e187b2db04094b"
I20260808 22:50:33.826660 29619 raft_consensus.cc:3037] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:33.826712 29170 raft_consensus.cc:3037] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:33.827487 29619 raft_consensus.cc:2454] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d2e352dc94c5dadd3f3a37af744dc in term 1.
I20260808 22:50:33.827634 29170 raft_consensus.cc:2454] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c8d2e352dc94c5dadd3f3a37af744dc in term 1.
I20260808 22:50:33.827661 29289 leader_election.cc:302] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c8d2e352dc94c5dadd3f3a37af744dc, d7a56a35f238414d931577d0248a2098; no voters:
I20260808 22:50:33.827732 29743 raft_consensus.cc:2781] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:33.827909 29743 raft_consensus.cc:686] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 1 LEADER]: Becoming Leader. State: Replica: 6c8d2e352dc94c5dadd3f3a37af744dc, State: Running, Role: LEADER
I20260808 22:50:33.828022 29743 consensus_queue.cc:237] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } }
I20260808 22:50:33.828840 28898 catalog_manager.cc:5495] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c8d2e352dc94c5dadd3f3a37af744dc (127.22.30.2). New cstate: current_term: 1 leader_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c8d2e352dc94c5dadd3f3a37af744dc" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44829 } health_report { overall_health: HEALTHY } } }
I20260808 22:50:33.854130 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.851762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.22.30.2:44829)
0808 22:50:33.851828 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:33.851830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:33.851838 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:33.851874 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:33.851875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:33.851913 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:33.851987 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:33.852135 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.852594 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.853347 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:33.853555 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:33.853630 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:33.853793 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:33.853884 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:33.853981 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:33.854016 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 22:50:33.854035 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:50:33.855703 29170 raft_consensus.cc:1261] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b [term 1 FOLLOWER]: Refusing update from remote peer 6c8d2e352dc94c5dadd3f3a37af744dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:33.856014 29619 raft_consensus.cc:1261] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 1 FOLLOWER]: Refusing update from remote peer 6c8d2e352dc94c5dadd3f3a37af744dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:33.856037 29743 consensus_queue.cc:1035] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f902c864bbbf4509a7e187b2db04094b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:33.856329 29743 consensus_queue.cc:1035] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a56a35f238414d931577d0248a2098" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 42951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 22:50:33.858392 29248 tablet.cc:2369] T c3518be9b55c4f03a29372b8d519b753 P f902c864bbbf4509a7e187b2db04094b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:33.858705 29754 mvcc.cc:205] Tried to move back new op lower bound from 7316395761070358528 to 7316395760960159744. Current Snapshot: MvccSnapshot[applied={T|T < 7316395761070358528}]
I20260808 22:50:33.860559 28898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:33.860658 28898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:33.861392 28898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aae257a0db8344bdbf35401e88ed555d: Sending DeleteTablet for 3 replicas of tablet c3518be9b55c4f03a29372b8d519b753
I20260808 22:50:33.861706 29587 tablet_service.cc:1514] Processing DeleteTablet for tablet c3518be9b55c4f03a29372b8d519b753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:33 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:35412
I20260808 22:50:33.861754 29135 tablet_service.cc:1514] Processing DeleteTablet for tablet c3518be9b55c4f03a29372b8d519b753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:33 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:34850
I20260808 22:50:33.862056 29365 tablet_service.cc:1514] Processing DeleteTablet for tablet c3518be9b55c4f03a29372b8d519b753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:33 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:37662
I20260808 22:50:33.862166 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 29022
I20260808 22:50:33.862308 29771 tablet_replica.cc:330] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: stopping tablet replica
I20260808 22:50:33.862430 29771 raft_consensus.cc:2229] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:33.862574 29771 raft_consensus.cc:2258] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:33.862735 29768 tablet_replica.cc:330] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: stopping tablet replica
I20260808 22:50:33.862843 29768 raft_consensus.cc:2229] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:33.862936 29768 raft_consensus.cc:2258] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:33.865024 29768 ts_tablet_manager.cc:1905] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:33.866739 29768 ts_tablet_manager.cc:1918] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:33.866827 29768 log.cc:1196] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/c3518be9b55c4f03a29372b8d519b753
I20260808 22:50:33.867101 29768 ts_tablet_manager.cc:1939] T c3518be9b55c4f03a29372b8d519b753 P d7a56a35f238414d931577d0248a2098: Deleting consensus metadata
I20260808 22:50:33.867671 28882 catalog_manager.cc:4841] TS d7a56a35f238414d931577d0248a2098 (127.22.30.3:42951): tablet c3518be9b55c4f03a29372b8d519b753 (table test-table [id=c9361464b8ce4bd082612fc408179390]) successfully deleted
I20260808 22:50:33.869777 29771 ts_tablet_manager.cc:1905] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:33.871279 29771 ts_tablet_manager.cc:1918] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:33.871354 29771 log.cc:1196] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/c3518be9b55c4f03a29372b8d519b753
I20260808 22:50:33.872375 29771 ts_tablet_manager.cc:1939] T c3518be9b55c4f03a29372b8d519b753 P 6c8d2e352dc94c5dadd3f3a37af744dc: Deleting consensus metadata
I20260808 22:50:33.872756 28882 catalog_manager.cc:4841] TS 6c8d2e352dc94c5dadd3f3a37af744dc (127.22.30.2:44829): tablet c3518be9b55c4f03a29372b8d519b753 (table test-table [id=c9361464b8ce4bd082612fc408179390]) successfully deleted
W20260808 22:50:33.876891 28882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:41173 (error 108)
I20260808 22:50:33.877087 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 29254
I20260808 22:50:33.878679 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:33.878530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:41173 (local address 127.0.0.1:34856)
0808 22:50:33.878587 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:33.878625 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:41173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 22:50:33.878793 28882 catalog_manager.cc:4584] TS f902c864bbbf4509a7e187b2db04094b (127.22.30.1:41173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3518be9b55c4f03a29372b8d519b753: Network error: Client connection negotiation failed: client connection to 127.22.30.1:41173: connect: Connection refused (error 111)
I20260808 22:50:33.885118 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 29482
I20260808 22:50:33.893658 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 28851
2026-08-08T22:50:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2172 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T22:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:33Z Disabled control of system clock
I20260808 22:50:33.928254 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43565
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:50:34.004492 29779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:34.004689 29779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:34.004711 29779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:34.005844 29779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:34.005882 29779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:34.005895 29779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:34.005908 29779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:34.005959 29779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:34.007223 29779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:34.007504 29779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:34.007788 29779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:34.012235 29788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.012235 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.012228 29791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.012574 29779 server_base.cc:1034] running on GCE node
I20260808 22:50:34.012821 29779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:34.013092 29779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:34.014227 29779 hybrid_clock.cc:648] HybridClock initialized: now 1786229434014216 us; error 30 us; skew 500 ppm
I20260808 22:50:34.015269 29779 webserver.cc:460] Webserver started at http://127.22.30.62:45721/ using document root <none> and password file <none>
I20260808 22:50:34.015530 29779 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:34.015574 29779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:34.015712 29779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:34.016623 29779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "2080c1792f5a414dbcf1c010e40bf41e"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "7f9eb49d4b59cdfe6761f0fdeb846e4e"
server_key_iv: "655832a7f839878a882536863e042d12"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.016956 29779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "2080c1792f5a414dbcf1c010e40bf41e"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "7f9eb49d4b59cdfe6761f0fdeb846e4e"
server_key_iv: "655832a7f839878a882536863e042d12"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.018144 29779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:34.018792 29808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.018944 29779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:34.019034 29779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "2080c1792f5a414dbcf1c010e40bf41e"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "7f9eb49d4b59cdfe6761f0fdeb846e4e"
server_key_iv: "655832a7f839878a882536863e042d12"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.019110 29779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:34.022073 29779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:34.022311 29779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:34.022440 29779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:34.026170 29779 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:43565
I20260808 22:50:34.026197 29897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:43565 every 8 connection(s)
I20260808 22:50:34.026468 29779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:34.026993 29898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:34.029440 29898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e: Bootstrap starting.
I20260808 22:50:34.029886 29898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:34.030089 29898 log.cc:824] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:34.030612 29898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e: No bootstrap required, opened a new log
I20260808 22:50:34.032037 29898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2080c1792f5a414dbcf1c010e40bf41e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43565 } }
I20260808 22:50:34.032147 29898 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:34.032167 29898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2080c1792f5a414dbcf1c010e40bf41e, State: Initialized, Role: FOLLOWER
I20260808 22:50:34.032300 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 29779
I20260808 22:50:34.032317 29898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2080c1792f5a414dbcf1c010e40bf41e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43565 } }
I20260808 22:50:34.032401 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:34.032413 29898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:34.032441 29898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:34.032487 29898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:34.032572 22648 external_mini_cluster.cc:1428] Setting key 55b49eb76173e7d44d4bdad7c1ae4464
I20260808 22:50:34.033146 29898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2080c1792f5a414dbcf1c010e40bf41e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43565 } }
I20260808 22:50:34.033241 29898 leader_election.cc:302] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2080c1792f5a414dbcf1c010e40bf41e; no voters:
I20260808 22:50:34.033373 29898 leader_election.cc:288] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:34.033425 29905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:34.033583 29905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [term 1 LEADER]: Becoming Leader. State: Replica: 2080c1792f5a414dbcf1c010e40bf41e, State: Running, Role: LEADER
I20260808 22:50:34.033605 29898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:34.033663 29905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2080c1792f5a414dbcf1c010e40bf41e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43565 } }
I20260808 22:50:34.033972 29905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2080c1792f5a414dbcf1c010e40bf41e. Latest consensus state: current_term: 1 leader_uuid: "2080c1792f5a414dbcf1c010e40bf41e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2080c1792f5a414dbcf1c010e40bf41e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43565 } } }
I20260808 22:50:34.034053 29905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:34.034157 29907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2080c1792f5a414dbcf1c010e40bf41e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2080c1792f5a414dbcf1c010e40bf41e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43565 } } }
I20260808 22:50:34.034222 29907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:34.034242 29912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:34.034705 29912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:34.035239 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.032994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.22.30.62:43565)
0808 22:50:34.033184 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.033192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.033220 (+ 28us) server_negotiation.cc:407] Connection header received
0808 22:50:34.033315 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.033321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.033363 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.033506 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.034582 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.034587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.034587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.034592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.034632 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.034645 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.034696 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.035006 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.035064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260808 22:50:34.039073 29912 catalog_manager.cc:1347] Generated new cluster ID: 2254720de13c463bba5ab97beb4e03d9
I20260808 22:50:34.039122 29912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:34.054625 29912 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:34.054795 29912 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:34.060683 29912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e: Generated new TSK 0
I20260808 22:50:34.060846 29912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:50:34.065651 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43565
--builtin_ntp_servers=127.22.30.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:50:34.144248 29937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:34.144497 29937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:34.144534 29937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:34.145660 29937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:34.145718 29937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:34.145759 29937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:34.147078 29937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:34.147330 29937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:34.147624 29937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:34.152421 29944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.152426 29948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.152429 29945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.152781 29937 server_base.cc:1034] running on GCE node
I20260808 22:50:34.152946 29937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:34.153117 29937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:34.154249 29937 hybrid_clock.cc:648] HybridClock initialized: now 1786229434154234 us; error 30 us; skew 500 ppm
I20260808 22:50:34.155350 29937 webserver.cc:460] Webserver started at http://127.22.30.1:33729/ using document root <none> and password file <none>
I20260808 22:50:34.155581 29937 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:34.155622 29937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:34.155728 29937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:34.156603 29937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "05a3b25e932744c28abad4e5f00c4bb6"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "5b72c36a1f16f3009a935ba1303c8678"
server_key_iv: "19a54b0f9b1070931af92a48c7d90616"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.156867 29937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "05a3b25e932744c28abad4e5f00c4bb6"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "5b72c36a1f16f3009a935ba1303c8678"
server_key_iv: "19a54b0f9b1070931af92a48c7d90616"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.158059 29937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:50:34.158702 29958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.158835 29937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:34.158910 29937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "05a3b25e932744c28abad4e5f00c4bb6"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "5b72c36a1f16f3009a935ba1303c8678"
server_key_iv: "19a54b0f9b1070931af92a48c7d90616"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.158957 29937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:34.161972 29937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:34.162142 29937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:34.162226 29937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:34.162355 29937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:34.162554 29937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:34.162578 29937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.162595 29937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:34.162606 29937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.167842 29937 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:35937
I20260808 22:50:34.167909 30137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:35937 every 8 connection(s)
I20260808 22:50:34.168201 29937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:34.170883 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.168879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:42739 (local address 127.22.30.62:43565)
0808 22:50:34.168939 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.168941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.169141 (+ 200us) server_negotiation.cc:407] Connection header received
0808 22:50:34.169604 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.169605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.169612 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.169676 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.170620 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.170621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.170621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.170626 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.170651 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.170655 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.170709 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.170805 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.170822 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:34.170945 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 29937
I20260808 22:50:34.171046 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:34.171033 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.168862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43565 (local address 127.22.30.1:42739)
0808 22:50:34.169044 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.169070 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.169488 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.169750 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.169754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.169762 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.170485 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.170486 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.170509 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.170532 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.170714 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.170718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.170721 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.170802 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.170855 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260808 22:50:34.171252 22648 external_mini_cluster.cc:1428] Setting key 7158e940353cd92ab0b9718b1a16ac52
I20260808 22:50:34.171644 30138 heartbeater.cc:347] Connected to a master server at 127.22.30.62:43565
I20260808 22:50:34.171718 30138 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:34.171868 30138 heartbeater.cc:511] Master 127.22.30.62:43565 requested a full tablet report, sending...
I20260808 22:50:34.172091 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43565
--builtin_ntp_servers=127.22.30.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 22:50:34.172349 29836 ts_manager.cc:194] Registered new tserver with Master: 05a3b25e932744c28abad4e5f00c4bb6 (127.22.30.1:35937)
W20260808 22:50:34.246757 30144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:34.246922 30144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:34.246943 30144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:34.248172 30144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:34.248214 30144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:34.248239 30144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:34.249576 30144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:34.249775 30144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:34.250054 30144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:34.254703 30152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.254695 30156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.254794 30151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.254902 30144 server_base.cc:1034] running on GCE node
I20260808 22:50:34.255086 30144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:34.255261 30144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:34.256397 30144 hybrid_clock.cc:648] HybridClock initialized: now 1786229434256387 us; error 38 us; skew 500 ppm
I20260808 22:50:34.257525 30144 webserver.cc:460] Webserver started at http://127.22.30.2:36881/ using document root <none> and password file <none>
I20260808 22:50:34.257738 30144 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:34.257778 30144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:34.257865 30144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:34.258818 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "37d4b156ff934efc8625fbc97dcd7168"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "ded26ba70b11978869b1cd6cd344c0a7"
server_key_iv: "40bdce9d3242ffb5136ed7f7b7dc1d25"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.259100 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "37d4b156ff934efc8625fbc97dcd7168"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "ded26ba70b11978869b1cd6cd344c0a7"
server_key_iv: "40bdce9d3242ffb5136ed7f7b7dc1d25"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.260305 30144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:34.260970 30167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.261198 30144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:34.261279 30144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "37d4b156ff934efc8625fbc97dcd7168"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "ded26ba70b11978869b1cd6cd344c0a7"
server_key_iv: "40bdce9d3242ffb5136ed7f7b7dc1d25"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.261329 30144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:34.264461 30144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:34.264744 30144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:34.264876 30144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:34.265049 30144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:34.265388 30144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:34.265424 30144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.265443 30144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:34.265460 30144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.271171 30144 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:44945
I20260808 22:50:34.271205 30342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:44945 every 8 connection(s)
I20260808 22:50:34.271543 30144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:34.274423 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.272182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:48231 (local address 127.22.30.62:43565)
0808 22:50:34.272240 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.272242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.272429 (+ 187us) server_negotiation.cc:407] Connection header received
0808 22:50:34.272871 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.272873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.272880 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.272929 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.274154 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.274155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.274156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.274161 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.274177 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.274181 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.274242 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.274339 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.274356 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:50:34.274634 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.272179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43565 (local address 127.22.30.2:48231)
0808 22:50:34.272332 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.272359 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.272752 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.272986 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.272989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.272998 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.274006 (+ 1008us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.274006 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.274032 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.274057 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.274240 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.274245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.274248 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.274346 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.274404 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260808 22:50:34.275079 30343 heartbeater.cc:347] Connected to a master server at 127.22.30.62:43565
I20260808 22:50:34.275162 30343 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:34.275269 30343 heartbeater.cc:511] Master 127.22.30.62:43565 requested a full tablet report, sending...
I20260808 22:50:34.275594 29836 ts_manager.cc:194] Registered new tserver with Master: 37d4b156ff934efc8625fbc97dcd7168 (127.22.30.2:44945)
I20260808 22:50:34.276714 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 30144
I20260808 22:50:34.276795 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:34.276911 22648 external_mini_cluster.cc:1428] Setting key f4f8418d213bbda2439be746f96eea8d
I20260808 22:50:34.277698 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43565
--builtin_ntp_servers=127.22.30.20:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 22:50:34.354581 30346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:34.354755 30346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:34.354774 30346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:34.355862 30346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:34.355899 30346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:34.355924 30346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:34.357127 30346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:34.357296 30346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:34.357506 30346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:34.362164 30358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.362162 30362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.362272 30360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.362519 30346 server_base.cc:1034] running on GCE node
I20260808 22:50:34.362751 30346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:34.362986 30346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:34.364149 30346 hybrid_clock.cc:648] HybridClock initialized: now 1786229434364137 us; error 24 us; skew 500 ppm
I20260808 22:50:34.365392 30346 webserver.cc:460] Webserver started at http://127.22.30.3:35935/ using document root <none> and password file <none>
I20260808 22:50:34.365615 30346 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:34.365682 30346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:34.365832 30346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:34.366843 30346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "363ec26d7dee4ff594b0e919dfedca2c"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "12e5645c6afb9b9028c34e66e886d4fb"
server_key_iv: "f9bdedb6a24ea9add85977d0090cdeab"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.367189 30346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "363ec26d7dee4ff594b0e919dfedca2c"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "12e5645c6afb9b9028c34e66e886d4fb"
server_key_iv: "f9bdedb6a24ea9add85977d0090cdeab"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.368497 30346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:34.369172 30370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.369323 30346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:34.369414 30346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "363ec26d7dee4ff594b0e919dfedca2c"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "12e5645c6afb9b9028c34e66e886d4fb"
server_key_iv: "f9bdedb6a24ea9add85977d0090cdeab"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.369491 30346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:34.372676 30346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:34.372912 30346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:34.373036 30346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:34.373236 30346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:34.373509 30346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:34.373558 30346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.373593 30346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:34.373620 30346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.379200 30346 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:45923
I20260808 22:50:34.379244 30559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:45923 every 8 connection(s)
I20260808 22:50:34.379588 30346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:34.382305 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.380250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:52271 (local address 127.22.30.62:43565)
0808 22:50:34.380325 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.380327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.380505 (+ 178us) server_negotiation.cc:407] Connection header received
0808 22:50:34.380934 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.380936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.380942 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.380989 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.382049 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.382050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.382050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.382054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.382069 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.382073 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.382124 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.382218 (+ 94us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.382235 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:50:34.382485 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.380206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43565 (local address 127.22.30.3:52271)
0808 22:50:34.380408 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.380435 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.380829 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.381046 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.381049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.381064 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.381900 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.381901 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.381925 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.381948 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.382128 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.382133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.382135 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.382223 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.382276 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260808 22:50:34.382565 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 30346
I20260808 22:50:34.382673 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:34.382831 22648 external_mini_cluster.cc:1428] Setting key 38cf4e7640d1b1ba02e9644cc2acfed1
I20260808 22:50:34.383263 30562 heartbeater.cc:347] Connected to a master server at 127.22.30.62:43565
I20260808 22:50:34.383390 30562 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:34.383548 30562 heartbeater.cc:511] Master 127.22.30.62:43565 requested a full tablet report, sending...
I20260808 22:50:34.383885 29835 ts_manager.cc:194] Registered new tserver with Master: 363ec26d7dee4ff594b0e919dfedca2c (127.22.30.3:45923)
I20260808 22:50:34.394411 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:50:34.396593 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.395693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.22.30.62:43565)
0808 22:50:34.395732 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.395734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.395852 (+ 118us) server_negotiation.cc:407] Connection header received
0808 22:50:34.395888 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.395888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.395892 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.395927 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.396388 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.396389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.396389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.396392 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.396405 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.396408 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.396448 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.396491 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.396502 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:34.398303 29835 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:34.399763 29835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:34.403103 30578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.401725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:44945 (local address 127.0.0.1:41356)
0808 22:50:34.401911 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.401946 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.401995 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.402251 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.402258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.402263 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.402742 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.402743 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.402756 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.402778 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.402978 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.402984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.402987 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.403028 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.403040 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":41,"threads_started":1}
I20260808 22:50:34.403357 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.401852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.22.30.2:44945)
0808 22:50:34.402109 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.402114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.402122 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:34.402145 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.402147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.402154 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.402248 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.402839 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.402846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.402846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.402850 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.402880 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.402905 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.403223 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.403252 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.403269 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20260808 22:50:34.403627 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.401956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:45923 (local address 127.0.0.1:36930)
0808 22:50:34.402171 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.402181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.402227 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.402689 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.402690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.402693 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.403258 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.403258 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.403264 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.403274 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.403508 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.403508 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.403509 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.403548 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.403560 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":61,"threads_started":1}
I20260808 22:50:34.403842 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.402023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.22.30.3:45923)
0808 22:50:34.402344 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.402347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.402354 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:34.402370 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.402371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.402377 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.402462 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.403349 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.403356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.403357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.403360 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.403407 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.403427 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.403484 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.403763 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.403778 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":74,"threads_started":1}
I20260808 22:50:34.403944 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.402145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.22.30.1:35937)
0808 22:50:34.402781 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.402786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.402795 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:34.402820 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.402822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.402830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.402910 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.403571 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.403579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.403580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.403584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.403615 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.403640 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.403697 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.403846 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.403861 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":77,"threads_started":1}
I20260808 22:50:34.403949 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.401765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35937 (local address 127.0.0.1:41538)
0808 22:50:34.402038 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.402047 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.402082 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.402909 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.402910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.402912 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.403468 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.403469 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.403476 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.403489 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.403747 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.403748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.403749 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.403822 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.403835 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":98,"threads_started":1}
I20260808 22:50:34.404009 30242 tablet_service.cc:1467] Processing CreateTablet for tablet eb336fde3b4e4ac7bc705f5552f72828 (DEFAULT_TABLE table=test-table [id=d95b2ce1d39545f8a18dfadb74b24fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:34.404392 30242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336fde3b4e4ac7bc705f5552f72828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:34.404649 30037 tablet_service.cc:1467] Processing CreateTablet for tablet eb336fde3b4e4ac7bc705f5552f72828 (DEFAULT_TABLE table=test-table [id=d95b2ce1d39545f8a18dfadb74b24fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:34.404970 30037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336fde3b4e4ac7bc705f5552f72828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:34.407575 30585 tablet_bootstrap.cc:492] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Bootstrap starting.
I20260808 22:50:34.407961 30452 tablet_service.cc:1467] Processing CreateTablet for tablet eb336fde3b4e4ac7bc705f5552f72828 (DEFAULT_TABLE table=test-table [id=d95b2ce1d39545f8a18dfadb74b24fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:34.408110 30587 tablet_bootstrap.cc:492] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: Bootstrap starting.
I20260808 22:50:34.408247 30585 tablet_bootstrap.cc:654] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:34.408279 30452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb336fde3b4e4ac7bc705f5552f72828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:34.408571 30585 log.cc:824] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:34.408643 30587 tablet_bootstrap.cc:654] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:34.408939 30587 log.cc:824] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:34.409354 30585 tablet_bootstrap.cc:492] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: No bootstrap required, opened a new log
I20260808 22:50:34.409410 30585 ts_tablet_manager.cc:1397] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:34.409716 30587 tablet_bootstrap.cc:492] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: No bootstrap required, opened a new log
I20260808 22:50:34.409781 30587 ts_tablet_manager.cc:1397] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:34.411209 30592 tablet_bootstrap.cc:492] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Bootstrap starting.
I20260808 22:50:34.411437 30587 raft_consensus.cc:348] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.411598 30587 raft_consensus.cc:374] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:34.411628 30587 raft_consensus.cc:729] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a3b25e932744c28abad4e5f00c4bb6, State: Initialized, Role: FOLLOWER
I20260808 22:50:34.411803 30592 tablet_bootstrap.cc:654] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:34.411746 30587 consensus_queue.cc:260] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.411739 30585 raft_consensus.cc:348] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.411897 30585 raft_consensus.cc:374] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:34.411926 30585 raft_consensus.cc:729] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d4b156ff934efc8625fbc97dcd7168, State: Initialized, Role: FOLLOWER
I20260808 22:50:34.412012 30585 consensus_queue.cc:260] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.412039 30587 ts_tablet_manager.cc:1428] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 22:50:34.412104 30592 log.cc:824] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:34.412191 30138 heartbeater.cc:503] Master 127.22.30.62:43565 was elected leader, sending a full tablet report...
I20260808 22:50:34.412262 30585 ts_tablet_manager.cc:1428] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:50:34.412350 30343 heartbeater.cc:503] Master 127.22.30.62:43565 was elected leader, sending a full tablet report...
I20260808 22:50:34.412915 30592 tablet_bootstrap.cc:492] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: No bootstrap required, opened a new log
I20260808 22:50:34.412979 30592 ts_tablet_manager.cc:1397] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:34.414666 30592 raft_consensus.cc:348] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.414769 30592 raft_consensus.cc:374] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:34.414789 30592 raft_consensus.cc:729] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363ec26d7dee4ff594b0e919dfedca2c, State: Initialized, Role: FOLLOWER
I20260808 22:50:34.414850 30592 consensus_queue.cc:260] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.415004 30592 ts_tablet_manager.cc:1428] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:34.415071 30562 heartbeater.cc:503] Master 127.22.30.62:43565 was elected leader, sending a full tablet report...
W20260808 22:50:34.418708 30141 tablet.cc:2369] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:34.514601 30600 raft_consensus.cc:482] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:34.514719 30600 raft_consensus.cc:504] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.515069 30600 leader_election.cc:288] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05a3b25e932744c28abad4e5f00c4bb6 (127.22.30.1:35937), 363ec26d7dee4ff594b0e919dfedca2c (127.22.30.3:45923)
I20260808 22:50:34.516479 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.515297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35937 (local address 127.22.30.2:56007)
0808 22:50:34.515354 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.515385 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.515424 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.515570 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.515572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.515574 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.516058 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.516058 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.516084 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.516099 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.516340 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.516341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.516342 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.516396 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.516409 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:50:34.516520 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.515389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:56007 (local address 127.22.30.1:35937)
0808 22:50:34.515432 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.515435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.515447 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:50:34.515475 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.515476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.515482 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.515561 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.516218 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.516220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.516221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.516226 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.516244 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.516249 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.516319 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.516408 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.516436 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:34.516584 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.515443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:45923 (local address 127.22.30.2:35447)
0808 22:50:34.515610 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.515620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.515665 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.515804 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.515805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.515807 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:34.516251 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:50:34.516251 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:34.516258 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:50:34.516270 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:34.516448 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:34.516449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:34.516449 (+ 0us) client_negotiation.cc:769] Sending connection context
0808 22:50:34.516496 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 22:50:34.516507 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
I20260808 22:50:34.516584 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.515573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:35447 (local address 127.22.30.3:45923)
0808 22:50:34.515614 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.515616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.515661 (+ 45us) server_negotiation.cc:407] Connection header received
0808 22:50:34.515716 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.515716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.515722 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.515792 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.516355 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.516356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.516357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.516362 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.516380 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.516385 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.516440 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.516496 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.516510 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:34.516820 30067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336fde3b4e4ac7bc705f5552f72828" candidate_uuid: "37d4b156ff934efc8625fbc97dcd7168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a3b25e932744c28abad4e5f00c4bb6" is_pre_election: true
I20260808 22:50:34.516865 30483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336fde3b4e4ac7bc705f5552f72828" candidate_uuid: "37d4b156ff934efc8625fbc97dcd7168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ec26d7dee4ff594b0e919dfedca2c" is_pre_election: true
I20260808 22:50:34.516987 30483 raft_consensus.cc:2454] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37d4b156ff934efc8625fbc97dcd7168 in term 0.
I20260808 22:50:34.516999 30067 raft_consensus.cc:2454] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37d4b156ff934efc8625fbc97dcd7168 in term 0.
I20260808 22:50:34.517191 30168 leader_election.cc:302] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363ec26d7dee4ff594b0e919dfedca2c, 37d4b156ff934efc8625fbc97dcd7168; no voters:
I20260808 22:50:34.517349 30600 raft_consensus.cc:2781] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:34.517416 30600 raft_consensus.cc:482] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:34.517449 30600 raft_consensus.cc:3037] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:34.518150 30600 raft_consensus.cc:504] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.518280 30600 leader_election.cc:288] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [CANDIDATE]: Term 1 election: Requested vote from peers 05a3b25e932744c28abad4e5f00c4bb6 (127.22.30.1:35937), 363ec26d7dee4ff594b0e919dfedca2c (127.22.30.3:45923)
I20260808 22:50:34.518467 30483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336fde3b4e4ac7bc705f5552f72828" candidate_uuid: "37d4b156ff934efc8625fbc97dcd7168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363ec26d7dee4ff594b0e919dfedca2c"
I20260808 22:50:34.518467 30067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb336fde3b4e4ac7bc705f5552f72828" candidate_uuid: "37d4b156ff934efc8625fbc97dcd7168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a3b25e932744c28abad4e5f00c4bb6"
I20260808 22:50:34.518545 30067 raft_consensus.cc:3037] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:34.518545 30483 raft_consensus.cc:3037] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:34.519428 30483 raft_consensus.cc:2454] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d4b156ff934efc8625fbc97dcd7168 in term 1.
I20260808 22:50:34.519443 30067 raft_consensus.cc:2454] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d4b156ff934efc8625fbc97dcd7168 in term 1.
I20260808 22:50:34.519572 30168 leader_election.cc:302] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a3b25e932744c28abad4e5f00c4bb6, 37d4b156ff934efc8625fbc97dcd7168; no voters:
I20260808 22:50:34.519644 30600 raft_consensus.cc:2781] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:34.519799 30600 raft_consensus.cc:686] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 1 LEADER]: Becoming Leader. State: Replica: 37d4b156ff934efc8625fbc97dcd7168, State: Running, Role: LEADER
I20260808 22:50:34.519896 30600 consensus_queue.cc:237] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } }
I20260808 22:50:34.520687 29835 catalog_manager.cc:5495] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37d4b156ff934efc8625fbc97dcd7168 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "37d4b156ff934efc8625fbc97dcd7168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37d4b156ff934efc8625fbc97dcd7168" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 44945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 } health_report { overall_health: UNKNOWN } } }
W20260808 22:50:34.521988 30344 tablet.cc:2369] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:34.549853 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.546028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.22.30.2:44945)
0808 22:50:34.546068 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.546070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.546076 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:34.546124 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.546124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.546134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.546180 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.549688 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.549690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.549690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.549693 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.549707 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.549711 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.549755 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.549797 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.549807 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:34.551477 30067 raft_consensus.cc:1261] T eb336fde3b4e4ac7bc705f5552f72828 P 05a3b25e932744c28abad4e5f00c4bb6 [term 1 FOLLOWER]: Refusing update from remote peer 37d4b156ff934efc8625fbc97dcd7168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:34.551488 30483 raft_consensus.cc:1261] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 1 FOLLOWER]: Refusing update from remote peer 37d4b156ff934efc8625fbc97dcd7168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:34.551771 30604 consensus_queue.cc:1035] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a3b25e932744c28abad4e5f00c4bb6" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:34.551888 30600 consensus_queue.cc:1035] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363ec26d7dee4ff594b0e919dfedca2c" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:34.557946 29835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:34.558041 29835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:34.558820 29835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2080c1792f5a414dbcf1c010e40bf41e: Sending DeleteTablet for 3 replicas of tablet eb336fde3b4e4ac7bc705f5552f72828
I20260808 22:50:34.559093 30037 tablet_service.cc:1514] Processing DeleteTablet for tablet eb336fde3b4e4ac7bc705f5552f72828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:34 UTC) from {username='slave'} at 127.0.0.1:41538
I20260808 22:50:34.559116 30242 tablet_service.cc:1514] Processing DeleteTablet for tablet eb336fde3b4e4ac7bc705f5552f72828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:34 UTC) from {username='slave'} at 127.0.0.1:41356
I20260808 22:50:34.559139 30452 tablet_service.cc:1514] Processing DeleteTablet for tablet eb336fde3b4e4ac7bc705f5552f72828 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:34 UTC) from {username='slave'} at 127.0.0.1:36930
I20260808 22:50:34.559402 30627 tablet_replica.cc:330] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: stopping tablet replica
I20260808 22:50:34.559450 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 29937
I20260808 22:50:34.559500 30627 raft_consensus.cc:2229] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:34.559588 30627 raft_consensus.cc:2258] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:34.559967 30627 ts_tablet_manager.cc:1905] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:34.561802 30627 ts_tablet_manager.cc:1918] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:34.561857 30627 log.cc:1196] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/eb336fde3b4e4ac7bc705f5552f72828
I20260808 22:50:34.562055 30627 ts_tablet_manager.cc:1939] T eb336fde3b4e4ac7bc705f5552f72828 P 363ec26d7dee4ff594b0e919dfedca2c: Deleting consensus metadata
I20260808 22:50:34.562355 29809 catalog_manager.cc:4841] TS 363ec26d7dee4ff594b0e919dfedca2c (127.22.30.3:45923): tablet eb336fde3b4e4ac7bc705f5552f72828 (table test-table [id=d95b2ce1d39545f8a18dfadb74b24fd9]) successfully deleted
I20260808 22:50:34.564000 30625 tablet_replica.cc:330] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: stopping tablet replica
I20260808 22:50:34.564096 30625 raft_consensus.cc:2229] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:34.564180 30625 raft_consensus.cc:2258] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:34.565037 30625 ts_tablet_manager.cc:1905] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:34.566566 30625 ts_tablet_manager.cc:1918] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:34.566653 30625 log.cc:1196] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/eb336fde3b4e4ac7bc705f5552f72828
I20260808 22:50:34.566890 30625 ts_tablet_manager.cc:1939] T eb336fde3b4e4ac7bc705f5552f72828 P 37d4b156ff934efc8625fbc97dcd7168: Deleting consensus metadata
I20260808 22:50:34.571648 29815 catalog_manager.cc:4841] TS 37d4b156ff934efc8625fbc97dcd7168 (127.22.30.2:44945): tablet eb336fde3b4e4ac7bc705f5552f72828 (table test-table [id=d95b2ce1d39545f8a18dfadb74b24fd9]) successfully deleted
W20260808 22:50:34.578191 29809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:35937 (error 108)
I20260808 22:50:34.578409 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 30144
I20260808 22:50:34.583113 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.581936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35937 (local address 127.0.0.1:41548)
0808 22:50:34.582986 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.583070 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":978}
W20260808 22:50:34.585379 29809 catalog_manager.cc:4584] TS 05a3b25e932744c28abad4e5f00c4bb6 (127.22.30.1:35937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb336fde3b4e4ac7bc705f5552f72828: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35937: connect: Connection refused (error 111)
I20260808 22:50:34.590098 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 30346
I20260808 22:50:34.597859 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 29779
2026-08-08T22:50:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (700 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 22:50:34.621089 22648 mini_oidc.cc:152] Starting JWKS server
I20260808 22:50:34.622257 22648 webserver.cc:460] Webserver started at https://127.0.0.1:37531/ using document root <none> and password file <none>
I20260808 22:50:34.622367 22648 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 22:50:34.622548 22648 webserver.cc:460] Webserver started at http://127.0.0.1:33053/ using document root <none> and password file <none>
2026-08-08T22:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:34Z Disabled control of system clock
I20260808 22:50:34.630889 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:46483
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:46483
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37531/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/testchainca.cert with env {}
W20260808 22:50:34.706672 30644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:34.706841 30644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:34.706861 30644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:34.707911 30644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:34.707943 30644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:34.707955 30644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:34.707967 30644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:34.707979 30644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:50:34.707991 30644 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:50:34.708002 30644 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37531/jwks.json
I20260808 22:50:34.709355 30644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:46483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37531/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:34.709519 30644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:34.709653 30644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:34.714136 30649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.714157 30650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.714221 30644 server_base.cc:1034] running on GCE node
W20260808 22:50:34.714157 30653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.714524 30644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:34.714787 30644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:34.715915 30644 hybrid_clock.cc:648] HybridClock initialized: now 1786229434715894 us; error 39 us; skew 500 ppm
I20260808 22:50:34.716974 30644 webserver.cc:460] Webserver started at http://127.22.30.62:33295/ using document root <none> and password file <none>
I20260808 22:50:34.717231 30644 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:34.717278 30644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:34.717401 30644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:34.718313 30644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "d94ba3c84b4449e08ae0af13172e98a5"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "b6f45f441a5c7322405d537647000290"
server_key_iv: "7e67a537b49f94d396688e75bfc0eac7"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.718659 30644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "d94ba3c84b4449e08ae0af13172e98a5"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "b6f45f441a5c7322405d537647000290"
server_key_iv: "7e67a537b49f94d396688e75bfc0eac7"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.720067 30644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:50:34.720758 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.720929 30644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:34.721027 30644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d94ba3c84b4449e08ae0af13172e98a5"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "b6f45f441a5c7322405d537647000290"
server_key_iv: "7e67a537b49f94d396688e75bfc0eac7"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.721107 30644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:34.742851 30644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:34.743114 30644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:34.743257 30644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:34.747030 30644 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:46483
I20260808 22:50:34.747067 30746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:46483 every 8 connection(s)
I20260808 22:50:34.747422 30644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:34.748396 30749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:34.750449 30749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5: Bootstrap starting.
I20260808 22:50:34.750990 30749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:34.751263 30749 log.cc:824] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:34.751909 30749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5: No bootstrap required, opened a new log
I20260808 22:50:34.753384 30749 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94ba3c84b4449e08ae0af13172e98a5" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46483 } }
I20260808 22:50:34.753509 30749 raft_consensus.cc:374] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:34.753557 30749 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94ba3c84b4449e08ae0af13172e98a5, State: Initialized, Role: FOLLOWER
I20260808 22:50:34.753695 30749 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94ba3c84b4449e08ae0af13172e98a5" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46483 } }
I20260808 22:50:34.753753 30749 raft_consensus.cc:388] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:34.753774 30749 raft_consensus.cc:482] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:34.753841 30749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:34.754330 30749 raft_consensus.cc:504] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94ba3c84b4449e08ae0af13172e98a5" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46483 } }
I20260808 22:50:34.754441 30749 leader_election.cc:302] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d94ba3c84b4449e08ae0af13172e98a5; no voters:
I20260808 22:50:34.754621 30749 leader_election.cc:288] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:34.754665 30761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:34.754835 30761 raft_consensus.cc:686] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [term 1 LEADER]: Becoming Leader. State: Replica: d94ba3c84b4449e08ae0af13172e98a5, State: Running, Role: LEADER
I20260808 22:50:34.754851 30749 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:34.754944 30761 consensus_queue.cc:237] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94ba3c84b4449e08ae0af13172e98a5" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46483 } }
I20260808 22:50:34.755434 30763 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d94ba3c84b4449e08ae0af13172e98a5. Latest consensus state: current_term: 1 leader_uuid: "d94ba3c84b4449e08ae0af13172e98a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94ba3c84b4449e08ae0af13172e98a5" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46483 } } }
I20260808 22:50:34.755535 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 30644
I20260808 22:50:34.755638 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:34.755632 30763 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:34.755862 22648 external_mini_cluster.cc:1428] Setting key 9cde756e307659086a77795c6d2a28ba
I20260808 22:50:34.755330 30762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d94ba3c84b4449e08ae0af13172e98a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d94ba3c84b4449e08ae0af13172e98a5" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46483 } } }
I20260808 22:50:34.756071 30762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:34.756147 30772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:34.756986 30772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:34.758895 30772 catalog_manager.cc:1347] Generated new cluster ID: 645b764982d04d0e97effb21dba05ae2
I20260808 22:50:34.758982 30772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:34.761065 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.756261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.22.30.62:46483)
0808 22:50:34.756453 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.756462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.756523 (+ 61us) server_negotiation.cc:407] Connection header received
0808 22:50:34.756710 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.756716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.756763 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.756917 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.757155 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.758464 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.759247 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.759715 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.759838 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:34.760493 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.760497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.760497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.760657 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.760686 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.760696 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.760767 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.760844 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.760916 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260808 22:50:34.761794 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:50:34.776918 30772 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:34.777379 30772 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:34.786024 30772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d94ba3c84b4449e08ae0af13172e98a5: Generated new TSK 0
I20260808 22:50:34.786183 30772 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 22:50:34.813237 30803 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 22:50:34.813366 30803 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 22:50:34.813602 30790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.798836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.22.30.62:46483)
0808 22:50:34.798890 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.798892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.799017 (+ 125us) server_negotiation.cc:407] Connection header received
0808 22:50:34.799081 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.799082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.799094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.799167 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:50:34.799303 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.800039 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.800683 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.800956 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.801047 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:34.801199 (+ 152us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:50:34.813375 (+ 12176us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 22:50:34.813542 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260808 22:50:34.813650 30790 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 22:50:34.814035 30802 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:34.798755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:46483 (local address 127.0.0.1:47346)
0808 22:50:34.798953 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:34.798963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:34.799016 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:34.799179 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:34.799180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:34.799181 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:50:34.799239 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:34.799240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.800125 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.800125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:34.800613 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:34.800614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.801053 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.801054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:34.801096 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:34.801112 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:50:34.813537 (+ 12425us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 22:50:34.813626 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.30.62:46483: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.30.62:46483: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37531/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 22:50:34.814494 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 30644
2026-08-08T22:50:34Z chronyd exiting
I20260808 22:50:34.858930 22648 test_util.cc:175] -----------------------------------------------
I20260808 22:50:34.859000 22648 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786229410580610-22648-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 22:50:34.859825 22648 mini_oidc.cc:152] Starting JWKS server
I20260808 22:50:34.860489 22648 webserver.cc:460] Webserver started at https://127.0.0.1:43177/ using document root <none> and password file <none>
I20260808 22:50:34.860528 22648 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 22:50:34.860705 22648 webserver.cc:460] Webserver started at http://127.0.0.1:46601/ using document root <none> and password file <none>
2026-08-08T22:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:34Z Disabled control of system clock
I20260808 22:50:34.868943 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43575
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43575
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43177/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/testchainca.cert with env {}
W20260808 22:50:34.945387 30811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:34.945595 30811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:34.945628 30811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:34.946699 30811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:34.946761 30811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:34.946781 30811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:34.946801 30811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:34.946856 30811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:50:34.946883 30811 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:50:34.946913 30811 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43177/jwks.json
I20260808 22:50:34.948450 30811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43177/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:34.948752 30811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:34.948957 30811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:34.953749 30811 server_base.cc:1034] running on GCE node
W20260808 22:50:34.953644 30820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.953647 30821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:34.953681 30827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:34.954087 30811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:34.954378 30811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:34.955533 30811 hybrid_clock.cc:648] HybridClock initialized: now 1786229434955520 us; error 41 us; skew 500 ppm
I20260808 22:50:34.956619 30811 webserver.cc:460] Webserver started at http://127.22.30.62:36903/ using document root <none> and password file <none>
I20260808 22:50:34.956862 30811 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:34.956930 30811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:34.957062 30811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:34.958055 30811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "d2c66ef4cb474b4690de7f47f94e9ce4"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "6cf9afce08428e37d6e6ec7b9356b374"
server_key_iv: "0665ad034a254e97f04c13cad204e8a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.958447 30811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "d2c66ef4cb474b4690de7f47f94e9ce4"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "6cf9afce08428e37d6e6ec7b9356b374"
server_key_iv: "0665ad034a254e97f04c13cad204e8a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.959827 30811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:34.960563 30835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:34.960706 30811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:34.960798 30811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d2c66ef4cb474b4690de7f47f94e9ce4"
format_stamp: "Formatted at 2026-08-08 22:50:34 on dist-test-slave-x4qh"
server_key: "6cf9afce08428e37d6e6ec7b9356b374"
server_key_iv: "0665ad034a254e97f04c13cad204e8a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:34.960871 30811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:34.975477 30811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:34.975749 30811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:34.975891 30811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:34.979573 30811 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:43575
I20260808 22:50:34.979614 30914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:43575 every 8 connection(s)
I20260808 22:50:34.979956 30811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:34.980597 30916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:34.983107 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 30811
I20260808 22:50:34.983224 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:34.983292 30916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4: Bootstrap starting.
I20260808 22:50:34.983474 22648 external_mini_cluster.cc:1428] Setting key 46d385e42268a41dfcccc651b97c995e
I20260808 22:50:34.984030 30916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:34.984404 30916 log.cc:824] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:34.985183 30916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4: No bootstrap required, opened a new log
I20260808 22:50:34.987344 30916 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43575 } }
I20260808 22:50:34.987531 30916 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:34.987576 30916 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c66ef4cb474b4690de7f47f94e9ce4, State: Initialized, Role: FOLLOWER
I20260808 22:50:34.987695 30916 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43575 } }
I20260808 22:50:34.987767 30916 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:34.987795 30916 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:34.987839 30916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:34.988524 30916 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43575 } }
I20260808 22:50:34.988641 30916 leader_election.cc:302] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c66ef4cb474b4690de7f47f94e9ce4; no voters:
I20260808 22:50:34.988791 30916 leader_election.cc:288] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:34.988816 30923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:34.988919 30923 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [term 1 LEADER]: Becoming Leader. State: Replica: d2c66ef4cb474b4690de7f47f94e9ce4, State: Running, Role: LEADER
I20260808 22:50:34.988997 30923 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43575 } }
I20260808 22:50:34.989040 30916 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:34.989289 30923 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c66ef4cb474b4690de7f47f94e9ce4. Latest consensus state: current_term: 1 leader_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43575 } } }
I20260808 22:50:34.989358 30923 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:34.989318 30925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c66ef4cb474b4690de7f47f94e9ce4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43575 } } }
I20260808 22:50:34.989542 30925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:34.989548 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:34.983952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.22.30.62:43575)
0808 22:50:34.984160 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:34.984169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:34.984208 (+ 39us) server_negotiation.cc:407] Connection header received
0808 22:50:34.984823 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:34.984832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:34.984920 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:34.985080 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:34.985269 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.986586 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.987336 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:34.987970 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:34.988096 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:34.988701 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:34.988705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:34.988705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:34.988907 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:34.988950 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:34.988966 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:34.989025 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:34.989231 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 22:50:34.989325 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260808 22:50:34.989619 30928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:34.990157 30928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:34.990463 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:50:34.992300 30928 catalog_manager.cc:1347] Generated new cluster ID: 93c4061f90024135962bf311ef09f6b9
I20260808 22:50:34.992362 30928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:35.001780 30928 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:35.002224 30928 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:35.005016 30928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2c66ef4cb474b4690de7f47f94e9ce4: Generated new TSK 0
I20260808 22:50:35.005220 30928 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 22:50:35.032269 30964 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 22:50:35.032341 30964 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 22:50:35.032579 30921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:35.012654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.22.30.62:43575)
0808 22:50:35.012695 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:35.012697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:35.012771 (+ 74us) server_negotiation.cc:407] Connection header received
0808 22:50:35.012824 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:35.012824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:35.012832 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:35.012885 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:50:35.013019 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.013441 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.014452 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.014620 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.014667 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.014840 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:50:35.032372 (+ 17532us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 22:50:35.032505 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 22:50:35.032637 30921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 22:50:35.032706 30963 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:35.012550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43575 (local address 127.0.0.1:40142)
0808 22:50:35.012699 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:35.012709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:35.012769 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:35.012902 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:35.012903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:35.012904 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:50:35.012956 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:35.012958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.013525 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.013526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:35.014375 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:35.014378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.014694 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.014695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:35.014752 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.014757 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:50:35.032489 (+ 17732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 22:50:35.032587 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.30.62:43575: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43177/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260808 22:50:35.032944 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 30811
2026-08-08T22:50:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 22:50:35.077739 22648 mini_oidc.cc:152] Starting JWKS server
I20260808 22:50:35.078352 22648 webserver.cc:460] Webserver started at https://127.0.0.1:37823/ using document root <none> and password file <none>
I20260808 22:50:35.078392 22648 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 22:50:35.078536 22648 webserver.cc:460] Webserver started at http://127.0.0.1:36003/ using document root <none> and password file <none>
2026-08-08T22:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:35Z Disabled control of system clock
I20260808 22:50:35.086830 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33683
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:33683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37823/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 22:50:35.160331 30972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:35.160490 30972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:35.160507 30972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:35.161470 30972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:35.161502 30972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:35.161514 30972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:35.161525 30972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:35.161536 30972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:50:35.161548 30972 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:50:35.161558 30972 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37823/jwks.json
I20260808 22:50:35.162688 30972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:33683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37823/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:35.162845 30972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:35.163039 30972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:35.167613 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:35.167629 30981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:35.167662 30979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:35.167798 30972 server_base.cc:1034] running on GCE node
I20260808 22:50:35.168026 30972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:35.168305 30972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:35.169442 30972 hybrid_clock.cc:648] HybridClock initialized: now 1786229435169432 us; error 27 us; skew 500 ppm
I20260808 22:50:35.170679 30972 webserver.cc:460] Webserver started at http://127.22.30.62:46011/ using document root <none> and password file <none>
I20260808 22:50:35.170913 30972 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:35.170956 30972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:35.171094 30972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:35.172068 30972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "e8f9c03a68d446fe8018b6b57c9e2373"
format_stamp: "Formatted at 2026-08-08 22:50:35 on dist-test-slave-x4qh"
server_key: "9b7660b66cc3432d707baef7c632bffb"
server_key_iv: "697c221885a3de3c17edb4f35bcd7e7f"
server_key_version: "encryptionkey@0"
I20260808 22:50:35.172369 30972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "e8f9c03a68d446fe8018b6b57c9e2373"
format_stamp: "Formatted at 2026-08-08 22:50:35 on dist-test-slave-x4qh"
server_key: "9b7660b66cc3432d707baef7c632bffb"
server_key_iv: "697c221885a3de3c17edb4f35bcd7e7f"
server_key_version: "encryptionkey@0"
I20260808 22:50:35.173458 30972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:35.174067 30993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:35.174229 30972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:35.174309 30972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "e8f9c03a68d446fe8018b6b57c9e2373"
format_stamp: "Formatted at 2026-08-08 22:50:35 on dist-test-slave-x4qh"
server_key: "9b7660b66cc3432d707baef7c632bffb"
server_key_iv: "697c221885a3de3c17edb4f35bcd7e7f"
server_key_version: "encryptionkey@0"
I20260808 22:50:35.174373 30972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:35.182837 30972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:35.183091 30972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:35.183223 30972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:35.187486 30972 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:33683
I20260808 22:50:35.187577 31077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:33683 every 8 connection(s)
I20260808 22:50:35.187880 30972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:35.188689 31079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:35.191190 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 30972
I20260808 22:50:35.191270 31079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373: Bootstrap starting.
I20260808 22:50:35.191288 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:35.191490 22648 external_mini_cluster.cc:1428] Setting key b15c4a9c46e969075a5184ddec1895d1
I20260808 22:50:35.191876 31079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:35.192204 31079 log.cc:824] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:35.192832 31079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373: No bootstrap required, opened a new log
I20260808 22:50:35.194360 31079 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33683 } }
I20260808 22:50:35.194496 31079 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:35.194540 31079 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f9c03a68d446fe8018b6b57c9e2373, State: Initialized, Role: FOLLOWER
I20260808 22:50:35.194643 31079 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33683 } }
I20260808 22:50:35.194736 31079 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:35.194783 31079 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:35.194828 31079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:35.195294 31079 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33683 } }
I20260808 22:50:35.195415 31079 leader_election.cc:302] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f9c03a68d446fe8018b6b57c9e2373; no voters:
I20260808 22:50:35.195566 31079 leader_election.cc:288] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:35.195602 31088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:35.195740 31088 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [term 1 LEADER]: Becoming Leader. State: Replica: e8f9c03a68d446fe8018b6b57c9e2373, State: Running, Role: LEADER
I20260808 22:50:35.195817 31079 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:35.195849 31088 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33683 } }
I20260808 22:50:35.196151 31088 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f9c03a68d446fe8018b6b57c9e2373. Latest consensus state: current_term: 1 leader_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33683 } } }
I20260808 22:50:35.196230 31088 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:35.196161 31091 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9c03a68d446fe8018b6b57c9e2373" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33683 } } }
I20260808 22:50:35.196348 31091 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:35.196425 31095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:35.196672 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:35.191990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.22.30.62:33683)
0808 22:50:35.192196 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:35.192204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:35.192233 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:50:35.192667 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:35.192673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:35.192722 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:35.192863 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:35.193060 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.193833 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.194551 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.195062 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.195194 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.195824 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:35.195828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:35.195828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:35.196018 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:35.196060 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:35.196075 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:35.196129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:35.196407 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 22:50:35.196492 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260808 22:50:35.196884 31095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:35.197533 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:50:35.199000 31095 catalog_manager.cc:1347] Generated new cluster ID: 27a3ed8062c640dd8823b797de17f4f8
I20260808 22:50:35.199057 31095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:35.203516 31095 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:35.204100 31095 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:35.210618 31095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f9c03a68d446fe8018b6b57c9e2373: Generated new TSK 0
I20260808 22:50:35.210772 31095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:50:35.224473 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:35.221111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.22.30.62:33683)
0808 22:50:35.221169 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:35.221171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:35.221268 (+ 97us) server_negotiation.cc:407] Connection header received
0808 22:50:35.221315 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:35.221316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:35.221326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:35.221383 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:50:35.221527 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.221991 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.223062 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.223223 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.223268 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.223461 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:50:35.224248 (+ 787us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37823/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 22:50:35.224390 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37823/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 22:50:35.224524 31083 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37823/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 22:50:35.224575 31130 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:35.220982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33683 (local address 127.0.0.1:60242)
0808 22:50:35.221186 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:35.221197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:35.221264 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:35.221400 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:35.221401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:35.221403 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:50:35.221460 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:35.221462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.222069 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.222070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:35.222983 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:35.222985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.223292 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.223293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:35.223351 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.223357 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:50:35.224365 (+ 1008us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37823/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 22:50:35.224480 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.30.62:33683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37823/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":125,"threads_started":1}
I20260808 22:50:35.224722 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 30972
E20260808 22:50:35.238641 31131 webserver.cc:533] Webserver: sslize failed:
E20260808 22:50:35.238701 31131 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T22:50:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (189 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T22:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:35Z Disabled control of system clock
I20260808 22:50:35.275240 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43245
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43245
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/testchainca.cert with env {}
W20260808 22:50:35.352329 31136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:35.352507 31136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:35.352528 31136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:35.353551 31136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:35.353585 31136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:35.353598 31136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:35.353610 31136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:35.353621 31136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 22:50:35.353633 31136 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 22:50:35.353644 31136 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 22:50:35.354966 31136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:35.355137 31136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:35.355280 31136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:35.360019 31147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:35.360140 31151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:35.360280 31146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:35.360549 31136 server_base.cc:1034] running on GCE node
I20260808 22:50:35.360787 31136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:35.361085 31136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:35.362252 31136 hybrid_clock.cc:648] HybridClock initialized: now 1786229435362239 us; error 26 us; skew 500 ppm
I20260808 22:50:35.363469 31136 webserver.cc:460] Webserver started at http://127.22.30.62:44501/ using document root <none> and password file <none>
I20260808 22:50:35.363715 31136 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:35.363792 31136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:35.363941 31136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:35.364964 31136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0"
format_stamp: "Formatted at 2026-08-08 22:50:35 on dist-test-slave-x4qh"
server_key: "085114b1984ce6fd09273b78626e51f7"
server_key_iv: "289f84bcaee64dddc913c4cb694fd024"
server_key_version: "encryptionkey@0"
I20260808 22:50:35.365342 31136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0"
format_stamp: "Formatted at 2026-08-08 22:50:35 on dist-test-slave-x4qh"
server_key: "085114b1984ce6fd09273b78626e51f7"
server_key_iv: "289f84bcaee64dddc913c4cb694fd024"
server_key_version: "encryptionkey@0"
I20260808 22:50:35.366663 31136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:35.367415 31162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:35.367599 31136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:35.367695 31136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0"
format_stamp: "Formatted at 2026-08-08 22:50:35 on dist-test-slave-x4qh"
server_key: "085114b1984ce6fd09273b78626e51f7"
server_key_iv: "289f84bcaee64dddc913c4cb694fd024"
server_key_version: "encryptionkey@0"
I20260808 22:50:35.367791 31136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:35.400542 31136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:35.400825 31136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:35.400978 31136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:35.404773 31136 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:43245
I20260808 22:50:35.404809 31243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:43245 every 8 connection(s)
I20260808 22:50:35.405117 31136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:35.406050 31245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:35.408226 31245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0: Bootstrap starting.
I20260808 22:50:35.408716 31245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:35.408944 31245 log.cc:824] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:35.409514 31245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0: No bootstrap required, opened a new log
I20260808 22:50:35.409832 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 31136
I20260808 22:50:35.409941 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:35.410112 22648 external_mini_cluster.cc:1428] Setting key 227b3e9bb266ccd7230d115248447bdd
I20260808 22:50:35.411520 31245 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43245 } }
I20260808 22:50:35.411638 31245 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:35.411657 31245 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5baadb9d4824e4f9831fbabdb3fc9b0, State: Initialized, Role: FOLLOWER
I20260808 22:50:35.411809 31245 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43245 } }
I20260808 22:50:35.411896 31245 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:35.411934 31245 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:35.411983 31245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:35.412451 31245 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43245 } }
I20260808 22:50:35.412561 31245 leader_election.cc:302] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5baadb9d4824e4f9831fbabdb3fc9b0; no voters:
I20260808 22:50:35.412717 31245 leader_election.cc:288] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:35.412766 31252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:35.412897 31252 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [term 1 LEADER]: Becoming Leader. State: Replica: d5baadb9d4824e4f9831fbabdb3fc9b0, State: Running, Role: LEADER
I20260808 22:50:35.412976 31245 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:35.412969 31252 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43245 } }
I20260808 22:50:35.414037 31254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5baadb9d4824e4f9831fbabdb3fc9b0. Latest consensus state: current_term: 1 leader_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43245 } } }
I20260808 22:50:35.414136 31254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:35.414182 31253 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5baadb9d4824e4f9831fbabdb3fc9b0" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43245 } } }
I20260808 22:50:35.414253 31253 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0 [sys.catalog]: This master's current role is: LEADER
W20260808 22:50:35.414997 31279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:50:35.415055 31279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:50:35.415095 31274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:35.415232 31274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:35.416383 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:35.410565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.22.30.62:43245)
0808 22:50:35.410743 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:35.410750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:35.410777 (+ 27us) server_negotiation.cc:407] Connection header received
0808 22:50:35.411500 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:35.411508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:35.411567 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:35.411736 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:35.411955 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.413413 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.414435 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.414978 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.415106 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.415728 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:35.415732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:35.415732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:50:35.415914 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:35.415949 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:50:35.415959 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:35.416032 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:35.416106 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 22:50:35.416177 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260808 22:50:35.416700 31274 catalog_manager.cc:1347] Generated new cluster ID: cb0db9e6afd14cc690f49d58f63981f8
I20260808 22:50:35.416742 31274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:35.417157 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:50:35.423026 31274 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:35.423553 31274 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:35.429098 31274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5baadb9d4824e4f9831fbabdb3fc9b0: Generated new TSK 0
I20260808 22:50:35.429243 31274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:50:36.711100 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:35.438395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.22.30.62:43245)
0808 22:50:35.438436 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:35.438437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:35.438537 (+ 100us) server_negotiation.cc:407] Connection header received
0808 22:50:35.438610 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:35.438611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:35.438620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:35.438675 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0808 22:50:35.438801 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.439223 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.440274 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.440454 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.440505 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.440738 (+ 233us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 22:50:36.710884 (+1270146us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 22:50:36.711042 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47068: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 22:50:36.711164 31250 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47068: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 22:50:36.711526 31292 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:35.438300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43245 (local address 127.0.0.1:47068)
0808 22:50:35.438466 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:35.438476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:35.438539 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:35.438691 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:35.438692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:35.438694 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 22:50:35.438742 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:35.438743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.439299 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.439300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:35.440200 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:35.440203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:35.440536 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:35.440538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:35.440632 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:35.440637 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 22:50:36.711118 (+1270481us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 22:50:36.711420 (+ 302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.30.62:43245: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260808 22:50:36.711901 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 31136
2026-08-08T22:50:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1468 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:36 dist-test-slave-x4qh krb5kdc[31308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:36 dist-test-slave-x4qh krb5kdc[31308](info): set up 2 sockets
Aug 08 22:50:36 dist-test-slave-x4qh krb5kdc[31308](info): commencing operation
krb5kdc: starting...
W20260808 22:50:38.061825 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:38.168454 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33423
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:38.242861 31327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:38.243053 31327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:38.243090 31327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:38.244154 31327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:38.244217 31327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:38.244243 31327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:38.244271 31327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:38.244288 31327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:38.245793 31327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:33423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:38.246026 31327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:38.246243 31327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:38.250695 31327 server_base.cc:1034] running on GCE node
W20260808 22:50:38.250660 31335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.250641 31340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.250675 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:38.251128 31327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:38.251446 31327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:38.252591 31327 hybrid_clock.cc:648] HybridClock initialized: now 1786229438252577 us; error 25 us; skew 500 ppm
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.254778 31327 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:38.255105 31327 webserver.cc:460] Webserver started at http://127.22.30.62:40721/ using document root <none> and password file <none>
I20260808 22:50:38.255290 31327 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:38.255324 31327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:38.255453 31327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:38.256228 31327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "97531b651d504577af02e72b5fed7443"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "715a5171c726d24f9fde79167ae603db"
server_key_iv: "e6cd9b11544213388816553f9c3a1f2b"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.256480 31327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "97531b651d504577af02e72b5fed7443"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "715a5171c726d24f9fde79167ae603db"
server_key_iv: "e6cd9b11544213388816553f9c3a1f2b"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.257571 31327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:38.258185 31353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.258349 31327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:38.258430 31327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "97531b651d504577af02e72b5fed7443"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "715a5171c726d24f9fde79167ae603db"
server_key_iv: "e6cd9b11544213388816553f9c3a1f2b"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.258481 31327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:38.267745 31327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:38.269249 31327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:38.269346 31327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:38.272725 31327 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:33423
I20260808 22:50:38.272794 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:33423 every 8 connection(s)
I20260808 22:50:38.273067 31327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:38.273808 31450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:38.275681 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443: Bootstrap starting.
I20260808 22:50:38.276095 31450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:38.276310 31450 log.cc:824] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:38.276840 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443: No bootstrap required, opened a new log
I20260808 22:50:38.278175 31450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97531b651d504577af02e72b5fed7443" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33423 } }
I20260808 22:50:38.278298 31450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:38.278321 31450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97531b651d504577af02e72b5fed7443, State: Initialized, Role: FOLLOWER
I20260808 22:50:38.278440 31450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97531b651d504577af02e72b5fed7443" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33423 } }
I20260808 22:50:38.278517 31450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:38.278538 31450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:38.278564 31450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:38.279006 31450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97531b651d504577af02e72b5fed7443" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33423 } }
I20260808 22:50:38.279078 31450 leader_election.cc:302] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97531b651d504577af02e72b5fed7443; no voters:
I20260808 22:50:38.279186 31450 leader_election.cc:288] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:38.279247 31459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:38.279452 31450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:38.279681 31459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [term 1 LEADER]: Becoming Leader. State: Replica: 97531b651d504577af02e72b5fed7443, State: Running, Role: LEADER
I20260808 22:50:38.279824 31459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97531b651d504577af02e72b5fed7443" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33423 } }
I20260808 22:50:38.280179 31460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97531b651d504577af02e72b5fed7443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97531b651d504577af02e72b5fed7443" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33423 } } }
I20260808 22:50:38.280193 31461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97531b651d504577af02e72b5fed7443. Latest consensus state: current_term: 1 leader_uuid: "97531b651d504577af02e72b5fed7443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97531b651d504577af02e72b5fed7443" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33423 } } }
I20260808 22:50:38.280321 31460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:38.280331 31461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:38.280673 31473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:38.281092 31473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:38.282286 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 31327
I20260808 22:50:38.282382 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:38.282368 31473 catalog_manager.cc:1347] Generated new cluster ID: 96baed36c1774035be047feb6d5fb87f
I20260808 22:50:38.282406 31473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:38.282557 22648 external_mini_cluster.cc:1428] Setting key 5b707b5bed0cf865b5f4533c50cc29f1
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:38.293466 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.282967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.22.30.62:33423)
0808 22:50:38.283091 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.283099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.283605 (+ 506us) server_negotiation.cc:407] Connection header received
0808 22:50:38.283724 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.283730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.283783 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.283903 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.284668 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.285972 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.286988 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.287538 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.287648 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.290167 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.290171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.290171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.290181 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.291921 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.292290 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.292293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.292293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.292390 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.292590 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.292591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.292592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.292892 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.293036 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.293132 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.293257 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260808 22:50:38.295408 31473 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:38.295835 31473 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:38.299666 31473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97531b651d504577af02e72b5fed7443: Generated new TSK 0
I20260808 22:50:38.299857 31473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:38.326056 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33423
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:38.403309 31492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:38.403554 31492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:38.403577 31492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:38.404695 31492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:38.404735 31492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:38.404760 31492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:38.406226 31492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:38.406486 31492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:38.406713 31492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:38.411602 31505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.411607 31502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.411616 31501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:38.411841 31492 server_base.cc:1034] running on GCE node
I20260808 22:50:38.412122 31492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:38.412379 31492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:38.413520 31492 hybrid_clock.cc:648] HybridClock initialized: now 1786229438413508 us; error 28 us; skew 500 ppm
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.415958 31492 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:38.416424 31492 webserver.cc:460] Webserver started at http://127.22.30.1:36861/ using document root <none> and password file <none>
I20260808 22:50:38.416656 31492 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:38.416723 31492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:38.416869 31492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:38.417848 31492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "8a8533ac897a4e7b978910d73bcc4c0d"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "149b975a7d08692110ec636f7453e838"
server_key_iv: "686ee668e156d72f3131b6a92731c0b0"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.418217 31492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "8a8533ac897a4e7b978910d73bcc4c0d"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "149b975a7d08692110ec636f7453e838"
server_key_iv: "686ee668e156d72f3131b6a92731c0b0"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.419513 31492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:38.420174 31515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.420360 31492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.420455 31492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "8a8533ac897a4e7b978910d73bcc4c0d"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "149b975a7d08692110ec636f7453e838"
server_key_iv: "686ee668e156d72f3131b6a92731c0b0"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.420542 31492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:38.445683 31492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:38.447388 31492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:38.447561 31492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:38.447801 31492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:38.448097 31492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:38.448153 31492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.448184 31492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:38.448210 31492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.454170 31492 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:36049
I20260808 22:50:38.454221 31692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:36049 every 8 connection(s)
I20260808 22:50:38.454555 31492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:38.461009 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 31492
I20260808 22:50:38.461086 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:38.461314 22648 external_mini_cluster.cc:1428] Setting key 3eb1bd705722430b3ac649455e79c212
I20260808 22:50:38.463657 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.455213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:36511 (local address 127.22.30.62:33423)
0808 22:50:38.455274 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.455276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.455479 (+ 203us) server_negotiation.cc:407] Connection header received
0808 22:50:38.456337 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.456338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.456348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.456402 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.456918 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.457392 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.458344 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.458536 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.458586 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.460644 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.460645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.460645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.460654 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.462232 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.462636 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.462637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.462638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.462700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.462954 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.462955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.462956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.463073 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.463170 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.463479 (+ 309us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.463566 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:50:38.463824 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.455165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33423 (local address 127.22.30.1:36511)
0808 22:50:38.455340 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:38.455401 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:38.456250 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:38.456451 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:38.456453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:38.456674 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:38.456847 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:38.456851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.457471 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.457472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:38.458267 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:38.458268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.458610 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.458611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:38.458703 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.459172 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:38.459179 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:38.460559 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:38.462318 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:38.462325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:38.462327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:38.462540 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:38.462770 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:38.462771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:38.462771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:38.462882 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:38.463163 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:38.463170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:38.463310 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 22:50:38.463454 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 22:50:38.463602 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260808 22:50:38.464295 31693 heartbeater.cc:347] Connected to a master server at 127.22.30.62:33423
I20260808 22:50:38.464401 31693 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:38.464582 31693 heartbeater.cc:511] Master 127.22.30.62:33423 requested a full tablet report, sending...
I20260808 22:50:38.465091 31386 ts_manager.cc:194] Registered new tserver with Master: 8a8533ac897a4e7b978910d73bcc4c0d (127.22.30.1:36049)
I20260808 22:50:38.465667 31386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:36511
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:38.486512 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33423
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:38.561110 31699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:38.561362 31699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:38.561396 31699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:38.562472 31699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:38.562547 31699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:38.562608 31699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:38.564069 31699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:38.564328 31699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:38.564568 31699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:38.569345 31707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.569358 31710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.569381 31706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:38.569630 31699 server_base.cc:1034] running on GCE node
I20260808 22:50:38.569831 31699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:38.570050 31699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:38.571223 31699 hybrid_clock.cc:648] HybridClock initialized: now 1786229438571212 us; error 28 us; skew 500 ppm
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.576473 31699 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:38.576893 31699 webserver.cc:460] Webserver started at http://127.22.30.2:43625/ using document root <none> and password file <none>
I20260808 22:50:38.577098 31699 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:38.577159 31699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:38.577288 31699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:38.578248 31699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "935798fc55294dadbe898f8fc9f4284e"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "e4ef65879847fbbd016253c4e62f3379"
server_key_iv: "d4e62e16a490917b70428f5403d8ddfd"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.578604 31699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "935798fc55294dadbe898f8fc9f4284e"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "e4ef65879847fbbd016253c4e62f3379"
server_key_iv: "d4e62e16a490917b70428f5403d8ddfd"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.579980 31699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:38.580713 31723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.580852 31699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:38.580930 31699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "935798fc55294dadbe898f8fc9f4284e"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "e4ef65879847fbbd016253c4e62f3379"
server_key_iv: "d4e62e16a490917b70428f5403d8ddfd"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.580976 31699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:38.598610 31699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:38.600131 31699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:38.600226 31699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:38.600383 31699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:38.600582 31699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:38.600610 31699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.600627 31699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:38.600638 31699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.606256 31699 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:40747
I20260808 22:50:38.606307 31901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:40747 every 8 connection(s)
I20260808 22:50:38.606578 31699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:38.611092 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 31699
I20260808 22:50:38.611214 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:38.611436 22648 external_mini_cluster.cc:1428] Setting key cec54fadb26dd1972b4879eecc051953
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:38.615470 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.607245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:36079 (local address 127.22.30.62:33423)
0808 22:50:38.607309 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.607311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.607492 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:50:38.608311 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.608312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.608322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.608377 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.608927 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.609406 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.610283 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.610444 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.610489 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.612596 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.612597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.612597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.612605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.614106 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.614552 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.614552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.614553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.614598 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.614856 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.614857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.614857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.614944 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.614989 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.615289 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.615348 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:50:38.615684 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.607190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33423 (local address 127.22.30.2:36079)
0808 22:50:38.607354 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:38.607413 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:38.608221 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:38.608436 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:38.608438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:38.608676 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:38.608855 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:38.608858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.609481 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.609482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:38.610217 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:38.610218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.610509 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.610510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:38.610595 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.610974 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:38.610981 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:38.612507 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:38.614188 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:38.614195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:38.614197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:38.614478 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:38.614659 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:38.614660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:38.614660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:38.614795 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:38.615017 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:38.615022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:38.615165 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 22:50:38.615268 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 22:50:38.615449 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260808 22:50:38.616155 31902 heartbeater.cc:347] Connected to a master server at 127.22.30.62:33423
I20260808 22:50:38.616258 31902 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:38.616417 31902 heartbeater.cc:511] Master 127.22.30.62:33423 requested a full tablet report, sending...
I20260808 22:50:38.616791 31386 ts_manager.cc:194] Registered new tserver with Master: 935798fc55294dadbe898f8fc9f4284e (127.22.30.2:40747)
I20260808 22:50:38.617151 31386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:36079
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:38.635859 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33423
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:38.712203 31907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:38.712373 31907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:38.712395 31907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:38.713538 31907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:38.713577 31907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:38.713600 31907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:38.714936 31907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:33423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:38.715111 31907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:38.715325 31907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:38.720086 31912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.720119 31919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:38.720135 31917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:38.720369 31907 server_base.cc:1034] running on GCE node
I20260808 22:50:38.720520 31907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:38.720718 31907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:38.721845 31907 hybrid_clock.cc:648] HybridClock initialized: now 1786229438721834 us; error 27 us; skew 500 ppm
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.724130 31907 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:38.724586 31907 webserver.cc:460] Webserver started at http://127.22.30.3:44133/ using document root <none> and password file <none>
I20260808 22:50:38.724756 31907 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:38.724803 31907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:38.724888 31907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:38.725787 31907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "603229411f2a4cb0a7b1cd38ea8c8ec6"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "5c7afe5392a52878645ac84a1400dff9"
server_key_iv: "b1c2c25ed14277c2931907231669a131"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.726056 31907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "603229411f2a4cb0a7b1cd38ea8c8ec6"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "5c7afe5392a52878645ac84a1400dff9"
server_key_iv: "b1c2c25ed14277c2931907231669a131"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.727202 31907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:50:38.727923 31927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.728088 31907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.728181 31907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "603229411f2a4cb0a7b1cd38ea8c8ec6"
format_stamp: "Formatted at 2026-08-08 22:50:38 on dist-test-slave-x4qh"
server_key: "5c7afe5392a52878645ac84a1400dff9"
server_key_iv: "b1c2c25ed14277c2931907231669a131"
server_key_version: "encryptionkey@0"
I20260808 22:50:38.728257 31907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:38.750748 31907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:38.753293 31907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:38.753453 31907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:38.753666 31907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:38.753945 31907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:38.753978 31907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.754026 31907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:38.754051 31907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:38.761425 31907 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:38085
I20260808 22:50:38.761483 32106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:38085 every 8 connection(s)
I20260808 22:50:38.761865 31907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:38.770466 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 31907
I20260808 22:50:38.770584 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:38.770784 22648 external_mini_cluster.cc:1428] Setting key 7650d479b88f02524e70e2603e2af5d3
I20260808 22:50:38.773491 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.762578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:43677 (local address 127.22.30.62:33423)
0808 22:50:38.762621 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.762624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.762873 (+ 249us) server_negotiation.cc:407] Connection header received
0808 22:50:38.763827 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.763829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.763844 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.763930 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.764558 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.765377 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.767044 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.767311 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.767347 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.770255 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.770256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.770257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.770264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.771998 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.772486 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.772487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.772488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.772544 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.772811 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.772812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.772812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.772930 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.773029 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.773300 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.773386 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:38.773662 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.762586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:33423 (local address 127.22.30.3:43677)
0808 22:50:38.762750 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:38.762784 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:38.763713 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:38.763978 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:38.763981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:38.764210 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:38.764466 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:38.764473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.765490 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.765491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:38.766951 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:38.766953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.767517 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.767518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:38.767645 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.768372 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:38.768381 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:38.770169 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:38.772067 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:38.772073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:38.772074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:38.772392 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:38.772613 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:38.772613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:38.772614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:38.772737 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:38.773032 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:38.773039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:38.773168 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 22:50:38.773273 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 22:50:38.773448 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260808 22:50:38.774075 32108 heartbeater.cc:347] Connected to a master server at 127.22.30.62:33423
I20260808 22:50:38.774199 32108 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:38.774402 32108 heartbeater.cc:511] Master 127.22.30.62:33423 requested a full tablet report, sending...
I20260808 22:50:38.774786 31386 ts_manager.cc:194] Registered new tserver with Master: 603229411f2a4cb0a7b1cd38ea8c8ec6 (127.22.30.3:38085)
I20260808 22:50:38.775152 31386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:43677
I20260808 22:50:38.782280 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/test-admin.keytab.
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.799197 22648 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 22:50:38.800897 32120 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:38.800115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.22.30.1:36049)
0808 22:50:38.800289 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.800293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.800335 (+ 42us) server_negotiation.cc:407] Connection header received
0808 22:50:38.800388 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.800390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.800404 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.800469 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.800800 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57184: BlockingRecv error: recv got EOF from 127.0.0.1:57184 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20260808 22:50:38.800942 22648 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc)
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.801848 22648 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.802627 22648 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:38.809883 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.803130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.22.30.1:36049)
0808 22:50:38.803171 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.803173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.803197 (+ 24us) server_negotiation.cc:407] Connection header received
0808 22:50:38.803282 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.803283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.803293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.803342 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.803749 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.804455 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.805155 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.805486 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.805582 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.807442 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.807449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.807450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.807456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.808928 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.809242 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.809244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.809245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.809306 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.809467 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.809467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.809467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.809579 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.809677 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.809756 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.809824 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 22:50:38.811842 32120 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:38.811165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.22.30.1:36049)
0808 22:50:38.811227 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.811229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.811276 (+ 47us) server_negotiation.cc:407] Connection header received
0808 22:50:38.811335 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.811336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.811345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.811421 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.811801 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57208: BlockingRecv error: recv got EOF from 127.0.0.1:57208 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 22:50:38.811910 22648 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc' of type FILE
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.812687 22648 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.813457 22648 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:38.819813 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.813930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.22.30.1:36049)
0808 22:50:38.813988 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.813990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.814034 (+ 44us) server_negotiation.cc:407] Connection header received
0808 22:50:38.814083 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.814084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.814093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.814139 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.814447 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.814841 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.815502 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.815703 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.815794 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.817603 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.817605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.817605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.817611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.818714 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.819051 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.819052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.819052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.819104 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.819299 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.819300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.819301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.819445 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.819540 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.819658 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.819725 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 22:50:38.821153 32120 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:38.820672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.22.30.1:36049)
0808 22:50:38.820705 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.820707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.820762 (+ 55us) server_negotiation.cc:407] Connection header received
0808 22:50:38.820826 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.820827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.820834 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.820883 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.821114 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57228: BlockingRecv error: recv got EOF from 127.0.0.1:57228 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 22:50:38.821224 22648 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786229410580610-22648-0/krb5kdc/krb5cc)
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.822027 22648 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:38.822901 22648 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[31308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.1@KRBTEST.COM
I20260808 22:50:38.829070 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:38.823346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.22.30.1:36049)
0808 22:50:38.823450 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:38.823451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:38.823458 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:38.823513 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:38.823514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:38.823520 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:38.823563 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:38.823917 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.824335 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.824985 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:38.825148 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:38.825193 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:38.826952 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:38.826953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:38.826953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:38.826958 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:38.828237 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.828506 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.828507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.828507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.828551 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:38.828690 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:38.828690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:38.828690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:38.828756 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:38.828816 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:38.828908 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 22:50:38.828954 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 22:50:38.829581 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 31492
I20260808 22:50:38.848507 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 31699
I20260808 22:50:38.856797 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 31907
I20260808 22:50:38.870489 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 31327
2026-08-08T22:50:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1986 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[32148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[32148](info): set up 2 sockets
Aug 08 22:50:38 dist-test-slave-x4qh krb5kdc[32148](info): commencing operation
krb5kdc: starting...
W20260808 22:50:40.041956 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab.
Entry for principal oryx/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab.
I20260808 22:50:40.145668 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37855
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab
--principal=oryx/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:40.221647 32167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:40.221844 32167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:40.221896 32167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:40.222929 32167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:40.222992 32167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:40.223016 32167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:40.223037 32167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:40.223059 32167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:40.224490 32167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab
--principal=oryx/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:40.224761 32167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:40.224977 32167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:40.229411 32173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.229475 32172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.229419 32176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:40.229593 32167 server_base.cc:1034] running on GCE node
I20260808 22:50:40.229841 32167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:40.230090 32167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:40.231221 32167 hybrid_clock.cc:648] HybridClock initialized: now 1786229440231213 us; error 24 us; skew 500 ppm
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:40.237366 32167 init.cc:377] Logged in from keytab as oryx/127.22.30.62@KRBTEST.COM (short username oryx)
I20260808 22:50:40.237824 32167 webserver.cc:460] Webserver started at http://127.22.30.62:42499/ using document root <none> and password file <none>
I20260808 22:50:40.238041 32167 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.238122 32167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.238272 32167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:40.239238 32167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "0a0929dc342b462a809e31b0c224cb5f"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "7dcf8a0a782939b163ae286a31171270"
server_key_iv: "2efa8c3bb71a515edaa7baef061e7984"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.239600 32167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "0a0929dc342b462a809e31b0c224cb5f"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "7dcf8a0a782939b163ae286a31171270"
server_key_iv: "2efa8c3bb71a515edaa7baef061e7984"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.241374 32167 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260808 22:50:40.242103 32184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.242271 32167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.242370 32167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "0a0929dc342b462a809e31b0c224cb5f"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "7dcf8a0a782939b163ae286a31171270"
server_key_iv: "2efa8c3bb71a515edaa7baef061e7984"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.242458 32167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:40.266265 32167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:40.267939 32167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:40.268112 32167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:40.272058 32167 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:37855
I20260808 22:50:40.272115 32265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:37855 every 8 connection(s)
I20260808 22:50:40.272401 32167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:40.273252 32267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:40.275202 32267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Bootstrap starting.
I20260808 22:50:40.275694 32267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:40.275902 32267 log.cc:824] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:40.276513 32267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: No bootstrap required, opened a new log
I20260808 22:50:40.277897 32267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } }
I20260808 22:50:40.278023 32267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:40.278044 32267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0929dc342b462a809e31b0c224cb5f, State: Initialized, Role: FOLLOWER
I20260808 22:50:40.278144 32267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } }
I20260808 22:50:40.278191 32267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:40.278211 32267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:40.278237 32267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:40.278937 32267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } }
I20260808 22:50:40.279029 32267 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0929dc342b462a809e31b0c224cb5f; no voters:
I20260808 22:50:40.279167 32267 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:40.279243 32271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:40.279430 32267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:40.279438 32271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 LEADER]: Becoming Leader. State: Replica: 0a0929dc342b462a809e31b0c224cb5f, State: Running, Role: LEADER
I20260808 22:50:40.279551 32271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } }
I20260808 22:50:40.279889 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 32167
I20260808 22:50:40.279881 32272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } }
I20260808 22:50:40.279954 32272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:40.279983 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:40.280189 22648 external_mini_cluster.cc:1428] Setting key 57e5a0205203139b498402401b3d385a
I20260808 22:50:40.280695 32274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0929dc342b462a809e31b0c224cb5f. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } }
I20260808 22:50:40.280862 32274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:40.280900 32286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:40.281531 32286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:40.283044 32286 catalog_manager.cc:1347] Generated new cluster ID: d3226eaeff6141db90d0a38c7a02a0c7
I20260808 22:50:40.283098 32286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:40.291404 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.280832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.22.30.62:37855)
0808 22:50:40.281010 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.281020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.281055 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:50:40.281159 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.281165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.281213 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.281355 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.282141 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.283348 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.284435 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.285024 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.285118 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.287683 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.287687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.287687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.287698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.289714 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.290078 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.290081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.290082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.290193 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.290501 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.290502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.290502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.290808 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.290957 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.291050 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.291184 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260808 22:50:40.292954 32286 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:40.293646 32286 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:40.303391 32286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Generated new TSK 0
I20260808 22:50:40.303586 32286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab.
Entry for principal oryx/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab.
I20260808 22:50:40.332391 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37855
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:40.409504 32308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:40.409679 32308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:40.409713 32308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:40.410761 32308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:40.410806 32308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:40.410830 32308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:40.412230 32308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:40.412456 32308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:40.412669 32308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:40.417245 32330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.417254 32328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.417254 32332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:40.417369 32308 server_base.cc:1034] running on GCE node
I20260808 22:50:40.417657 32308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:40.417862 32308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:40.419013 32308 hybrid_clock.cc:648] HybridClock initialized: now 1786229440419005 us; error 31 us; skew 500 ppm
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:40.421449 32308 init.cc:377] Logged in from keytab as oryx/127.22.30.1@KRBTEST.COM (short username oryx)
I20260808 22:50:40.421893 32308 webserver.cc:460] Webserver started at http://127.22.30.1:43089/ using document root <none> and password file <none>
I20260808 22:50:40.422134 32308 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.422199 32308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.422335 32308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:40.423277 32308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "547b73eca41143cc8446da2fdb59e031"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "a35e7afc384fe6f3e805f4cfb7571aee"
server_key_iv: "45c151d35619242b406e6e8d26bab791"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.423686 32308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "547b73eca41143cc8446da2fdb59e031"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "a35e7afc384fe6f3e805f4cfb7571aee"
server_key_iv: "45c151d35619242b406e6e8d26bab791"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.425002 32308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:40.425733 32340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.425908 32308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:40.426003 32308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "547b73eca41143cc8446da2fdb59e031"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "a35e7afc384fe6f3e805f4cfb7571aee"
server_key_iv: "45c151d35619242b406e6e8d26bab791"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.426079 32308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:40.431990 32308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:40.433498 32308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:40.433655 32308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:40.433844 32308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:40.434168 32308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:40.434223 32308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.434262 32308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:40.434299 32308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.439821 32308 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:33815
I20260808 22:50:40.439890 32517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:33815 every 8 connection(s)
I20260808 22:50:40.440173 32308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:40.446717 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 32308
I20260808 22:50:40.446822 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:40.447041 22648 external_mini_cluster.cc:1428] Setting key 897450d61265ccd9c22fdee59d7d30c4
I20260808 22:50:40.449062 32524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.440833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:48695)
0808 22:50:40.440991 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.441022 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.441444 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.441668 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.441671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.442189 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.442354 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.442358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.442987 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.442987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.443833 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.443836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.444180 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.444181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.444268 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.444801 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.444815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.446200 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.447834 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.447840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.447841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.448041 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.448261 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.448261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.448261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.448353 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.448626 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.448630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.448727 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.448810 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.448916 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260808 22:50:40.449103 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.440847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:48695 (local address 127.22.30.62:37855)
0808 22:50:40.440896 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.440898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.441123 (+ 225us) server_negotiation.cc:407] Connection header received
0808 22:50:40.441522 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.441523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.441536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.441607 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.442428 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.442901 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.443908 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.444105 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.444161 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.446278 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.446279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.446279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.446287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.447737 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.448122 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.448123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.448124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.448194 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.448417 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.448418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.448418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.448561 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.448632 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.448858 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.448947 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:40.449970 32521 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37855
I20260808 22:50:40.450066 32521 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:40.450214 32521 heartbeater.cc:511] Master 127.22.30.62:37855 requested a full tablet report, sending...
I20260808 22:50:40.450691 32209 ts_manager.cc:194] Registered new tserver with Master: 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815)
I20260808 22:50:40.451261 32209 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:48695
WARNING: no policy specified for oryx/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab.
Entry for principal oryx/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab.
I20260808 22:50:40.471192 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37855
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:40.546221 32528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:40.546391 32528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:40.546412 32528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:40.547493 32528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:40.547531 32528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:40.547556 32528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:40.549067 32528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:40.549335 32528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:40.549547 32528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:40.554301 32528 server_base.cc:1034] running on GCE node
W20260808 22:50:40.554278 32536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.554265 32541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.554272 32537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:40.554684 32528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:40.554919 32528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:40.556082 32528 hybrid_clock.cc:648] HybridClock initialized: now 1786229440556070 us; error 27 us; skew 500 ppm
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:40.558526 32528 init.cc:377] Logged in from keytab as oryx/127.22.30.2@KRBTEST.COM (short username oryx)
I20260808 22:50:40.558946 32528 webserver.cc:460] Webserver started at http://127.22.30.2:37693/ using document root <none> and password file <none>
I20260808 22:50:40.559183 32528 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.559250 32528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.559438 32528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:40.560420 32528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "60aff0300d014752891613db853120bd"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "13f97e860e142eed2d48e1be064519f3"
server_key_iv: "dac1deeabe61a20aca940f646fbb9a17"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.560768 32528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "60aff0300d014752891613db853120bd"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "13f97e860e142eed2d48e1be064519f3"
server_key_iv: "dac1deeabe61a20aca940f646fbb9a17"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.561980 32528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:40.562666 32551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.562863 32528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:40.562960 32528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "60aff0300d014752891613db853120bd"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "13f97e860e142eed2d48e1be064519f3"
server_key_iv: "dac1deeabe61a20aca940f646fbb9a17"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.563041 32528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:40.572189 32528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:40.573767 32528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:40.573909 32528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:40.574115 32528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:40.574406 32528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:40.574465 32528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.574501 32528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:40.574525 32528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.580170 32528 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:36137
I20260808 22:50:40.580209 32752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:36137 every 8 connection(s)
I20260808 22:50:40.580462 32528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:40.585477 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 32528
I20260808 22:50:40.585616 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:40.585810 22648 external_mini_cluster.cc:1428] Setting key 39d354ac243e04c70762cb942c6f33d9
I20260808 22:50:40.589709 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.581073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:44341 (local address 127.22.30.62:37855)
0808 22:50:40.581130 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.581131 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.581283 (+ 152us) server_negotiation.cc:407] Connection header received
0808 22:50:40.581677 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.581678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.581687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.581740 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.582510 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.582968 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.583888 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.584056 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.584109 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.586164 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.586166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.586167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.586174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.588359 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.588761 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.588763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.588763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.588829 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.589080 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.589081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.589081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.589184 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.589276 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.589536 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.589626 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:50:40.589881 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.581039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:44341)
0808 22:50:40.581181 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.581213 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.581583 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.581804 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.581807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.582284 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.582437 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.582441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.583044 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.583045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.583819 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.583821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.584124 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.584125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.584206 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.584618 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.584625 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.586061 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.588458 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.588464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.588466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.588670 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.588898 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.588899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.588900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.589008 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.589277 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.589284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.589404 (+ 120us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.589502 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.589642 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260808 22:50:40.590365 32753 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37855
I20260808 22:50:40.590480 32753 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:40.590643 32753 heartbeater.cc:511] Master 127.22.30.62:37855 requested a full tablet report, sending...
I20260808 22:50:40.591045 32209 ts_manager.cc:194] Registered new tserver with Master: 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:40.591444 32209 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:44341
WARNING: no policy specified for oryx/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab.
Entry for principal oryx/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab.
I20260808 22:50:40.610946 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37855
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:40.688329 32762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:40.688501 32762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:40.688522 32762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:40.689509 32762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:40.689541 32762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:40.689564 32762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:40.690742 32762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:40.690888 32762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:40.691023 32762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:40.695660 302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.695717 307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.695777 303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:40.695817 32762 server_base.cc:1034] running on GCE node
I20260808 22:50:40.696146 32762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:40.696333 32762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:40.697518 32762 hybrid_clock.cc:648] HybridClock initialized: now 1786229440697503 us; error 27 us; skew 500 ppm
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:40.699978 32762 init.cc:377] Logged in from keytab as oryx/127.22.30.3@KRBTEST.COM (short username oryx)
I20260808 22:50:40.700448 32762 webserver.cc:460] Webserver started at http://127.22.30.3:44349/ using document root <none> and password file <none>
I20260808 22:50:40.700660 32762 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.700726 32762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.700879 32762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:40.701906 32762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "a3d920d87053407ab5264d8ea4ea160e"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "00937c502f42a38c1ad2d8cd5f0e20ed"
server_key_iv: "837a2ebe11c5bab7af0bdf790c98ed43"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.702234 32762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "a3d920d87053407ab5264d8ea4ea160e"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "00937c502f42a38c1ad2d8cd5f0e20ed"
server_key_iv: "837a2ebe11c5bab7af0bdf790c98ed43"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.703490 32762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:40.704201 319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.704386 32762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:40.704479 32762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "a3d920d87053407ab5264d8ea4ea160e"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "00937c502f42a38c1ad2d8cd5f0e20ed"
server_key_iv: "837a2ebe11c5bab7af0bdf790c98ed43"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.704558 32762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:40.716470 32762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:40.718209 32762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:40.718369 32762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:40.718575 32762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:40.718869 32762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:40.718937 32762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.718971 32762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:40.718995 32762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.724741 32762 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:36781
I20260808 22:50:40.724792 505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:36781 every 8 connection(s)
I20260808 22:50:40.725064 32762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:40.733152 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.725678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:34207 (local address 127.22.30.62:37855)
0808 22:50:40.725751 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.725754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.725930 (+ 176us) server_negotiation.cc:407] Connection header received
0808 22:50:40.726359 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.726360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.726370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.726428 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.727254 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.727814 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.728911 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.729123 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.729174 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.730968 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.730969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.730970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.730975 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.732108 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.732434 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.732434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.732435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.732479 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.732677 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.732678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.732678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.732763 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.732828 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.733038 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.733094 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:50:40.733357 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.725648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:34207)
0808 22:50:40.725817 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.725847 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.726256 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.726483 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.726486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.727000 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.727171 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.727175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.727895 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.727896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.728791 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.728794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.729196 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.729197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.729333 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.729688 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.729695 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.730891 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.732174 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.732180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.732182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.732372 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.732533 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.732534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.732534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.732626 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.732834 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.732838 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.732936 (+ 98us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.733009 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.733128 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":96,"threads_started":1}
I20260808 22:50:40.733918 507 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37855
I20260808 22:50:40.734010 507 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:40.734220 507 heartbeater.cc:511] Master 127.22.30.62:37855 requested a full tablet report, sending...
I20260808 22:50:40.734596 32209 ts_manager.cc:194] Registered new tserver with Master: a3d920d87053407ab5264d8ea4ea160e (127.22.30.3:36781)
I20260808 22:50:40.734982 32209 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:34207
I20260808 22:50:40.735249 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 32762
I20260808 22:50:40.735374 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:40.735608 22648 external_mini_cluster.cc:1428] Setting key 2ab9567a056889a630f8f2e775240ac7
I20260808 22:50:40.736440 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.30.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.61.keytab.
Entry for principal oryx/127.22.30.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.61.keytab.
I20260808 22:50:40.760350 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:37385
--webserver_interface=127.22.30.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.61.keytab
--principal=oryx/127.22.30.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37855,127.22.30.61:37385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:40.849642 513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:40.849822 513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:40.849843 513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:40.850821 513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:40.850852 513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:40.850864 513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:40.850875 513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:40.850886 513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:40.852403 513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37855,127.22.30.61:37385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.61.keytab
--principal=oryx/127.22.30.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.30.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
W20260808 22:50:40.852500 513 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.30.62:37855,127.22.30.61:37385'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 22:50:40.852591 513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:40.852737 513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:40.857885 524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.857872 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.858058 521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:40.858191 513 server_base.cc:1034] running on GCE node
I20260808 22:50:40.858372 513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:40.858616 513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:40.859778 513 hybrid_clock.cc:648] HybridClock initialized: now 1786229440859763 us; error 26 us; skew 500 ppm
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:40.862540 513 init.cc:377] Logged in from keytab as oryx/127.22.30.61@KRBTEST.COM (short username oryx)
I20260808 22:50:40.862987 513 webserver.cc:460] Webserver started at http://127.22.30.61:43141/ using document root <none> and password file <none>
I20260808 22:50:40.863163 513 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.863200 513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.863282 513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:40.864522 513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/instance:
uuid: "91f6e29cfed14feda53fb4a725048b9d"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "30386677484788d14ee82407b17f32a2"
server_key_iv: "67bfbf4f2b4986326b76728016e953a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.864832 513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal/instance:
uuid: "91f6e29cfed14feda53fb4a725048b9d"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "30386677484788d14ee82407b17f32a2"
server_key_iv: "67bfbf4f2b4986326b76728016e953a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.866096 513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:50:40.866744 533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.866968 513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:40.867064 513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
uuid: "91f6e29cfed14feda53fb4a725048b9d"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "30386677484788d14ee82407b17f32a2"
server_key_iv: "67bfbf4f2b4986326b76728016e953a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.867146 513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:40.879223 513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:40.881197 513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:40.881384 513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:40.889424 513 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.61:37385
I20260808 22:50:40.889456 624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.61:37385 every 8 connection(s)
I20260808 22:50:40.889775 513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
I20260808 22:50:40.890563 625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:40.892575 625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:50:40.894834 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 513
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.61@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
I20260808 22:50:40.902282 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.893194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:48654)
0808 22:50:40.893436 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.893465 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.893977 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.894166 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.894169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.894396 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.894572 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.894576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.895204 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.895205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.896475 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.896478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.896960 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.896960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.897046 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.897506 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.897513 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.899163 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.900737 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.900744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.900746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.900972 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.901219 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.901220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.901220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.901317 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.901671 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.901677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.901815 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.901915 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.902063 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":164,"threads_started":1}
I20260808 22:50:40.902385 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.893259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.22.30.62:37855)
0808 22:50:40.893325 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.893327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.893537 (+ 210us) server_negotiation.cc:407] Connection header received
0808 22:50:40.894056 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.894057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.894066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.894112 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.894635 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.895121 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.896593 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.896880 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.896950 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.899280 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.899282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.899282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.899290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.900641 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.901071 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.901072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.901072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.901145 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.901404 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.901405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.901406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.901591 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.901661 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.901950 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.902261 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:50:40.903013 625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:50:40.904983 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.895293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.22.30.61:37385)
0808 22:50:40.895554 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.895558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.895576 (+ 18us) server_negotiation.cc:407] Connection header received
0808 22:50:40.895607 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.895608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.895625 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.895689 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.896365 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.897575 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.898490 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.898960 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.899017 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.901140 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.901148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.901148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.901155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.903033 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.903536 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.903540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.903540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.903607 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.903875 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.903876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.903876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.904031 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.904215 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.904741 (+ 526us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.904816 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":186,"threads_started":1}
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.61@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
I20260808 22:50:40.916436 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.903236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53816)
0808 22:50:40.903284 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.903292 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.903327 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.910035 (+ 6708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.910036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.910205 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.910264 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.910267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.911145 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.911146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.911714 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.911716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.912115 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.912116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.912178 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.912610 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.912615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.913805 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.915277 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.915278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.915278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.915511 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.915742 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.915743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.915743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.915797 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.916098 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.916099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.916139 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.916224 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.916328 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 22:50:40.916504 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.903549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.22.30.61:37385)
0808 22:50:40.905049 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.905051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.905059 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:40.905076 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.905093 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.905102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.905156 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.910365 (+ 5209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.911017 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.911814 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.912008 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.912116 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.913930 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.913931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.913931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.913938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.915173 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.915609 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.915610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.915610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.915664 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.915872 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.915873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.915873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.915972 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.916103 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.916260 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.916383 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":1488,"thread_start_us":88,"threads_started":1}
I20260808 22:50:40.917927 625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Bootstrap starting.
I20260808 22:50:40.918992 625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:40.919695 625 log.cc:824] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:40.920491 625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: No bootstrap required, opened a new log
I20260808 22:50:40.922839 625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } }
I20260808 22:50:40.922950 625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:40.923012 625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f6e29cfed14feda53fb4a725048b9d, State: Initialized, Role: FOLLOWER
I20260808 22:50:40.923139 625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } }
I20260808 22:50:40.923518 625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:40.923537 637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } } }
I20260808 22:50:40.923640 637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:50:40.925877 660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:50:40.925927 660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:50:40.927443 513 master_runner.cc:428] Detected that this master 91f6e29cfed14feda53fb4a725048b9d is joining an existing cluster
I20260808 22:50:40.927515 513 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.30.61:37385
I20260808 22:50:40.935156 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.928991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.22.30.62:37855)
0808 22:50:40.929099 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.929102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.929216 (+ 114us) server_negotiation.cc:407] Connection header received
0808 22:50:40.929276 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.929276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.929285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.929356 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.929791 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.930229 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.931174 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.931479 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.931542 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.932643 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.932644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.932645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.932651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.934036 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934421 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.934422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.934422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.934471 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934655 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.934656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.934656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.934761 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.934859 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.935004 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.935077 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260808 22:50:40.935353 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.928923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53828)
0808 22:50:40.929496 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.929504 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.929555 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.929704 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.929705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.929880 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.929935 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.929936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.930709 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.930710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.931500 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.931503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.931950 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.931951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.932015 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.932448 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.932453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.932800 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.934440 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.934441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.934617 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.934788 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934788 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.934789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.934855 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.935097 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.935098 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.935127 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.935178 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.935269 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260808 22:50:40.935171 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.928899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:48670)
0808 22:50:40.929148 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.929157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.929212 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.929362 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.929362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.929589 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.929720 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.929722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.930332 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.930333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.931085 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.931087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.931552 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.931553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.931666 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.932077 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.932083 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.932553 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.934133 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.934135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.934351 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.934534 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.934535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.934586 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.934853 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.934853 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.934892 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.934955 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.935063 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260808 22:50:40.935683 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.929083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.22.30.61:37385)
0808 22:50:40.929117 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.929118 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.929548 (+ 430us) server_negotiation.cc:407] Connection header received
0808 22:50:40.929624 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.929624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.929633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.929694 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.930009 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.930614 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.931601 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.931842 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.931897 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.932879 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.932880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.932881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.932885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.934370 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934678 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.934679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.934679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.934718 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.934900 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.934900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.934901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.935021 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.935075 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.935535 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.935614 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.62@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
I20260808 22:50:40.943682 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.937303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.22.30.61:37385)
0808 22:50:40.937334 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.937335 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.937495 (+ 160us) server_negotiation.cc:407] Connection header received
0808 22:50:40.937578 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.937579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.937587 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.937628 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.938021 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.938444 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.939176 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.939355 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.939451 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.941353 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.941354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.941355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.941360 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.942584 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.943010 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.943010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.943010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.943053 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.943235 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.943236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.943236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.943323 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.943443 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.943581 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.943636 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:40.943768 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.937253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53844)
0808 22:50:40.937384 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.937415 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.937509 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.937631 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.937634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.937898 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.937938 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.937939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.938527 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.938529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.939093 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.939095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.939457 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.939458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.939544 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.939920 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.939924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.941245 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.942662 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.942668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.942670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.942925 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.943113 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.943113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.943114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.943178 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.943397 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.943404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.943469 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.943560 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.943666 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260808 22:50:40.944389 32209 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.30.61:37385: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d"
member_type: NON_VOTER
last_known_addr {
host: "127.22.30.61"
port: 37385
}
attrs {
promote: true
}
}
dest_uuid: "0a0929dc342b462a809e31b0c224cb5f"
cas_config_opid_index: -1
I20260808 22:50:40.944729 32209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } }
I20260808 22:50:40.945163 675 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:40.945465 580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:40.946132 580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Refusing update from remote peer 0a0929dc342b462a809e31b0c224cb5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 22:50:40.946147 675 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.30.61:37385
I20260808 22:50:40.946237 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:40.946324 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:40.946421 513 master.cc:537] Master@127.22.30.61:37385 shutting down...
I20260808 22:50:40.946805 674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 22:50:40.947465 32188 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 22:50:40.949199 513 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:40.949283 513 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:40.949327 513 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: stopping tablet replica
I20260808 22:50:40.951696 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.951612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53846)
0808 22:50:40.951652 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.951679 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 22:50:40.951807 32188 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f -> Peer 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61:37385): Couldn't send request to peer 91f6e29cfed14feda53fb4a725048b9d. Status: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 22:50:40.961884 513 master.cc:552] Master@127.22.30.61:37385 shutdown complete.
I20260808 22:50:40.961951 513 master_runner.cc:305] Clearing existing system tablet
I20260808 22:50:40.962072 513 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.962127 513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.962666 513 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260808 22:50:40.963079 680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.963191 513 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:40.963244 513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.963264 513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
uuid: "91f6e29cfed14feda53fb4a725048b9d"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "30386677484788d14ee82407b17f32a2"
server_key_iv: "67bfbf4f2b4986326b76728016e953a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.963924 513 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:40.965216 513 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 22:50:40.965307 513 log.cc:1196] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 22:50:40.965514 513 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Deleting consensus metadata
I20260808 22:50:40.965642 513 master_runner.cc:315] Copying system tablet from 127.22.30.62:37855
I20260808 22:50:40.966239 513 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: tablet copy: Beginning tablet copy session from remote peer at address 127.22.30.62:37855
I20260808 22:50:40.972397 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.966553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:48686)
0808 22:50:40.966679 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:40.966692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:40.966741 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:40.966852 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:40.966852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:40.966983 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:40.967321 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.967323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.968249 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.968250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.968796 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:40.968798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.969237 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.969238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:40.969287 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.969696 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:40.969700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:40.969994 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:40.971624 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.971625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.971625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.971770 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.971961 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:40.971961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:40.971961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:40.971999 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:40.972189 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:40.972189 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:40.972222 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 22:50:40.972268 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 22:50:40.972335 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260808 22:50:40.972434 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:40.966596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.22.30.62:37855)
0808 22:50:40.966632 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:40.966634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:40.966751 (+ 117us) server_negotiation.cc:407] Connection header received
0808 22:50:40.966778 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:40.966779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:40.966789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:40.966849 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:40.967421 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.968162 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.968873 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:40.969165 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:40.969233 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:40.970078 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:40.970079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:40.970080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:40.970087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:40.971559 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.971849 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.971850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.971851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.971903 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:40.972031 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:40.972031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:40.972031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:40.972135 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:40.972191 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:40.972293 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 22:50:40.972359 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:40.972805 32240 tablet_copy_service.cc:140] P 0a0929dc342b462a809e31b0c224cb5f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91f6e29cfed14feda53fb4a725048b9d ({username='oryx', principal='oryx/127.22.30.61@KRBTEST.COM'} at 127.0.0.1:48686)
I20260808 22:50:40.972918 32240 tablet_copy_service.cc:161] P 0a0929dc342b462a809e31b0c224cb5f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91f6e29cfed14feda53fb4a725048b9d at {username='oryx', principal='oryx/127.22.30.61@KRBTEST.COM'} at 127.0.0.1:48686: session id = 91f6e29cfed14feda53fb4a725048b9d-00000000000000000000000000000000
I20260808 22:50:40.973760 32240 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 22:50:40.974303 513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 22:50:40.975085 513 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: tablet copy: Starting download of 0 data blocks...
I20260808 22:50:40.975229 513 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: tablet copy: Starting download of 1 WAL segments...
I20260808 22:50:40.976138 513 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 22:50:40.976827 32240 tablet_copy_service.cc:342] P 0a0929dc342b462a809e31b0c224cb5f: Request end of tablet copy session 91f6e29cfed14feda53fb4a725048b9d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.30.61@KRBTEST.COM'} at 127.0.0.1:48686
I20260808 22:50:40.976888 32240 tablet_copy_service.cc:433] P 0a0929dc342b462a809e31b0c224cb5f: ending tablet copy session 91f6e29cfed14feda53fb4a725048b9d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91f6e29cfed14feda53fb4a725048b9d
I20260808 22:50:40.979266 513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:40.980564 691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.980576 693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:40.980593 698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:40.980935 513 server_base.cc:1034] running on GCE node
I20260808 22:50:40.981024 513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:40.981156 513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:40.982290 513 hybrid_clock.cc:648] HybridClock initialized: now 1786229440982281 us; error 35 us; skew 500 ppm
Aug 08 22:50:40 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:40.983353 513 init.cc:377] Logged in from keytab as oryx/127.22.30.61@KRBTEST.COM (short username oryx)
I20260808 22:50:40.983629 513 webserver.cc:460] Webserver started at http://127.22.30.61:35893/ using document root <none> and password file <none>
I20260808 22:50:40.983736 513 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:40.983791 513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:40.984323 513 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:40.984802 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:40.984918 513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:40.984964 513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
uuid: "91f6e29cfed14feda53fb4a725048b9d"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "30386677484788d14ee82407b17f32a2"
server_key_iv: "67bfbf4f2b4986326b76728016e953a3"
server_key_version: "encryptionkey@0"
I20260808 22:50:40.985008 513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.000120 513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:41.001616 513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:41.004675 513 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.61:37385
I20260808 22:50:41.004846 792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.61:37385 every 8 connection(s)
I20260808 22:50:41.005016 513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
I20260808 22:50:41.005720 795 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:50:41.006122 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Bootstrap starting.
I20260808 22:50:41.008476 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:50:41.008708 795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: Bootstrap complete.
I20260808 22:50:41.008927 795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } }
I20260808 22:50:41.009003 795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91f6e29cfed14feda53fb4a725048b9d, State: Initialized, Role: LEARNER
I20260808 22:50:41.009029 795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } }
I20260808 22:50:41.009212 795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:41.009222 798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:41.009341 798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: LEARNER
I20260808 22:50:41.010665 815 catalog_manager.cc:1259] Loaded cluster ID: d3226eaeff6141db90d0a38c7a02a0c7
I20260808 22:50:41.010707 815 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: loading cluster ID for follower catalog manager: success
I20260808 22:50:41.011809 815 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: acquiring CA information for follower catalog manager: success
I20260808 22:50:41.012060 815 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:50:41.018312 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.011479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.22.30.61:37385)
0808 22:50:41.012163 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.012166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.012174 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:41.012194 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.012194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.012203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.012265 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.012907 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.013472 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.014490 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.014697 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.014746 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.015975 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.015976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.015977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.015982 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.017261 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.017624 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.017625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.017626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.017672 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.017881 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.017882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.017882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.017969 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.018036 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.018190 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.018249 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":35,"threads_started":1}
WARNING: no policy specified for oryx/127.22.30.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.60.keytab.
Entry for principal oryx/127.22.30.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.60.keytab.
I20260808 22:50:41.042400 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:35099
--webserver_interface=127.22.30.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.60.keytab
--principal=oryx/127.22.30.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:41.120240 819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:41.120395 819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:41.120414 819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:41.121433 819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:41.121474 819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:41.121488 819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:41.121543 819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:41.121568 819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:41.122991 819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.60.keytab
--principal=oryx/127.22.30.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.30.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:41.123215 819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:41.123433 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:50:41.127970 819 server_base.cc:1034] running on GCE node
W20260808 22:50:41.127873 834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.127866 833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.127873 837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.128335 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:41.128598 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:41.129736 819 hybrid_clock.cc:648] HybridClock initialized: now 1786229441129723 us; error 33 us; skew 500 ppm
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:41.131928 819 init.cc:377] Logged in from keytab as oryx/127.22.30.60@KRBTEST.COM (short username oryx)
I20260808 22:50:41.132295 819 webserver.cc:460] Webserver started at http://127.22.30.60:46867/ using document root <none> and password file <none>
I20260808 22:50:41.132505 819 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:41.132562 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:41.132706 819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:41.133574 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/instance:
uuid: "8a0d6b9c718243d0a13144d55681ea16"
format_stamp: "Formatted at 2026-08-08 22:50:41 on dist-test-slave-x4qh"
server_key: "53a966a4a2776be51b84468e98c1e00d"
server_key_iv: "d79d5ed60c70c843b5cd0a31fb9f853e"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.133884 819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal/instance:
uuid: "8a0d6b9c718243d0a13144d55681ea16"
format_stamp: "Formatted at 2026-08-08 22:50:41 on dist-test-slave-x4qh"
server_key: "53a966a4a2776be51b84468e98c1e00d"
server_key_iv: "d79d5ed60c70c843b5cd0a31fb9f853e"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.134985 819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:50:41.135648 847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.135810 819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:41.135892 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
uuid: "8a0d6b9c718243d0a13144d55681ea16"
format_stamp: "Formatted at 2026-08-08 22:50:41 on dist-test-slave-x4qh"
server_key: "53a966a4a2776be51b84468e98c1e00d"
server_key_iv: "d79d5ed60c70c843b5cd0a31fb9f853e"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.135977 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.168361 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:41.169978 819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:41.170121 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:41.175017 819 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.60:35099
I20260808 22:50:41.175045 923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.60:35099 every 8 connection(s)
I20260808 22:50:41.175361 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
I20260808 22:50:41.176344 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:41.176993 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 819
I20260808 22:50:41.177857 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.60@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.30.60@KRBTEST.COM
I20260808 22:50:41.187314 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.178832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.22.30.62:37855)
0808 22:50:41.178873 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.178875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.178951 (+ 76us) server_negotiation.cc:407] Connection header received
0808 22:50:41.179077 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.179078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.179091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.179154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.179655 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.180413 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.181572 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.181851 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.181912 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.183944 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.183945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.183945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.183952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.185783 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.186303 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.186305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.186305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.186363 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.186644 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.186645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.186646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.186797 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.186867 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.187148 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.187226 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:41.187474 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.178664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:48698)
0808 22:50:41.178848 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.178888 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.178951 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.179183 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.179185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.179487 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.179586 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.179590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.180508 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.180509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.181487 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.181489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.181956 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.181957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.182024 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.182346 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.182355 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.183856 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.185906 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.185912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.185914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.186237 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.186445 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.186446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.186446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.186576 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.186876 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.186883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.187030 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.187103 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.187233 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":17,"thread_start_us":101,"threads_started":1}
I20260808 22:50:41.187666 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.177548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.22.30.60:35099)
0808 22:50:41.177698 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.177705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.177726 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:50:41.178113 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.178118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.178158 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.178321 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.178945 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.180312 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.181308 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.181692 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.181823 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.184502 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.184510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.184510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.184516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.186402 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.186749 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.186751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.186751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.186812 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.187000 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.187000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.187000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.187143 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.187276 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.187363 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.187469 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260808 22:50:41.188104 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.60@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
I20260808 22:50:41.193117 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.188344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53870)
0808 22:50:41.188376 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.188387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.188427 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.188593 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.188593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.188697 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.188738 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.188739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.189294 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.189295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.189714 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.189715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.190010 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.190010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.190055 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.190410 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.190415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.191361 (+ 946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.192484 (+ 1123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.192485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.192485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.192616 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.192757 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.192758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.192758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.192801 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.192949 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.192949 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.192976 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.193018 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.193073 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 22:50:41.193132 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.188432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.22.30.61:37385)
0808 22:50:41.188499 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.188501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.188509 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:41.188533 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.188533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.188548 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.188591 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.188782 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.189231 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.189779 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.189955 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.189999 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.191465 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.191466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.191466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.191474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.192425 (+ 951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.192673 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.192674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.192674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.192712 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.192834 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.192834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.192834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.192907 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.192944 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.193030 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.193082 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:50:41.193569 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.60@KRBTEST.COM for oryx/127.22.30.60@KRBTEST.COM
I20260808 22:50:41.198995 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.193755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.0.0.1:38304)
0808 22:50:41.193798 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.193806 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.193838 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.193942 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.193943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.194050 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.194072 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.194073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.194610 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.194610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.195068 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.195070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.195464 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.195465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.195502 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.195960 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.195969 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.197019 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.198224 (+ 1205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.198224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.198225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.198384 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.198549 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.198549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.198549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.198598 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.198786 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.198787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.198817 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.198866 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.198934 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 22:50:41.199054 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.193796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.22.30.60:35099)
0808 22:50:41.193839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.193841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.193848 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:41.193882 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.193883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.193889 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.193938 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.194116 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.194560 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.195156 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.195358 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.195465 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.197087 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.197088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.197088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.197093 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.198154 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.198447 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.198448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.198448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.198493 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.198650 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.198650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.198650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.198733 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.198782 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.198881 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.198988 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":27}
I20260808 22:50:41.200124 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Bootstrap starting.
I20260808 22:50:41.200841 924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:41.201092 924 log.cc:824] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:41.201718 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: No bootstrap required, opened a new log
I20260808 22:50:41.203014 924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } }
I20260808 22:50:41.203109 924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:41.203128 924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0d6b9c718243d0a13144d55681ea16, State: Initialized, Role: FOLLOWER
I20260808 22:50:41.203262 924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } }
I20260808 22:50:41.203575 924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:41.203549 932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } } }
I20260808 22:50:41.203630 932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:50:41.205638 946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:50:41.205720 946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:50:41.207742 819 master_runner.cc:428] Detected that this master 8a0d6b9c718243d0a13144d55681ea16 is joining an existing cluster
I20260808 22:50:41.207841 819 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.30.60:35099
I20260808 22:50:41.238914 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 32308
I20260808 22:50:41.242626 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.209364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.22.30.62:37855)
0808 22:50:41.209796 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.209798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.209936 (+ 138us) server_negotiation.cc:407] Connection header received
0808 22:50:41.210084 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.210084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.210093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.210116 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.210544 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.211176 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.212203 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.212458 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.212497 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.213470 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.213471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.213472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.213478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.215221 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.241829 (+ 26608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.241830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.241831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.241883 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.242078 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.242078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.242079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.242199 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.242240 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.242440 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.242520 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260808 22:50:41.243165 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.209431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:48706)
0808 22:50:41.209886 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.209895 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.209985 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.210208 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.210209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.210403 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.210466 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.210468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.211257 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.211259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.212116 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.212119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.212586 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.212587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.212640 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.213027 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.213032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.213366 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.215305 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.215306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.215306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.241621 (+ 26315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.241957 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.241958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.241958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.242009 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.242285 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.242286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.242327 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.242939 (+ 612us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.243037 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":270,"threads_started":1}
I20260808 22:50:41.245077 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.210829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.22.30.61:37385)
0808 22:50:41.210866 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.210868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.210875 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:41.210891 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.210892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.210899 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.210945 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.211410 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.212052 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.213065 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.213325 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.213365 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.242198 (+ 28833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.242199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.242199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.242206 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.243801 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.244165 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.244166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.244166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.244217 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.244481 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.244482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.244483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.244615 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.244662 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.244889 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.244977 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:41.245790 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.209306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53886)
0808 22:50:41.209512 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.209521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.209633 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.210984 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.210984 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.211221 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.211301 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.211302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.212140 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.212141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.212949 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.212952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.213449 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.213450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.213938 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.241754 (+ 27816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.241758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.242115 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.243892 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.243892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.243893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.244085 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.244298 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.244299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.244299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.244392 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.244713 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.244714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.244761 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.245350 (+ 589us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.245682 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":37,"thread_start_us":77,"threads_started":1}
I20260808 22:50:41.246898 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.209867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.22.30.60:35099)
0808 22:50:41.209895 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.209896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.210968 (+ 1072us) server_negotiation.cc:407] Connection header received
0808 22:50:41.211040 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.211040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.211046 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.211067 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.211498 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.212101 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.213069 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.213298 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.213330 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.214515 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.214516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.214516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.214522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.216104 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.246103 (+ 29999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.246104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.246104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.246156 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.246385 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.246385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.246386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.246500 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.246544 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.246730 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.246804 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 22:50:41.247319 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.209932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.0.0.1:38314)
0808 22:50:41.210913 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.210922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.210986 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.211125 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.211126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.211323 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.211423 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.211425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.212181 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.212181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.212977 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.212979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.213406 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.213407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.213459 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.214104 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.214108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.214442 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.216186 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.216187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.216187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.245594 (+ 29407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.246222 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.246222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.246223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.246305 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.246593 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.246594 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.246623 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.247146 (+ 523us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.247222 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":24,"thread_start_us":72,"threads_started":1}
I20260808 22:50:41.248597 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:33815
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:37855
--webserver_port=43089
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229440, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.62@KRBTEST.COM for oryx/127.22.30.60@KRBTEST.COM
I20260808 22:50:41.253692 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.246008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.22.30.60:35099)
0808 22:50:41.246467 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.246468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.246475 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:41.246504 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.246504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.246513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.246933 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.246967 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.247871 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.248633 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.248961 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.249037 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.250909 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.250910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.250911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.250916 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.252482 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.252896 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.252897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.252897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.252958 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.253171 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.253171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.253172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.253293 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.253357 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.253533 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.253609 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":67,"threads_started":1}
I20260808 22:50:41.253707 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.245634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.0.0.1:38330)
0808 22:50:41.245676 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.245687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.245742 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.246612 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.246612 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.246803 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.246856 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.246858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.247975 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.247976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.248557 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.248559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.249052 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.249053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.249132 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.249483 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.249487 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.250823 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.252571 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.252572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.252572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.252816 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.253033 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.253034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.253034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.253100 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.253363 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.253364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.253411 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.253485 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.253584 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260808 22:50:41.254362 32209 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.30.60:35099: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16"
member_type: NON_VOTER
last_known_addr {
host: "127.22.30.60"
port: 35099
}
attrs {
promote: true
}
}
dest_uuid: "0a0929dc342b462a809e31b0c224cb5f"
cas_config_opid_index: 5
I20260808 22:50:41.254596 32209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } }
I20260808 22:50:41.254978 675 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.255466 885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:41.256510 885 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Refusing update from remote peer 0a0929dc342b462a809e31b0c224cb5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 22:50:41.256807 674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:41.257120 675 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.30.60:35099
I20260808 22:50:41.257189 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.257278 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.258185 885 raft_consensus.cc:858] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } }
W20260808 22:50:41.258256 885 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } }
I20260808 22:50:41.258960 885 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:41.259948 972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0929dc342b462a809e31b0c224cb5f. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:41.260082 932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:41.260143 932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 22:50:41.260160 819 master.cc:537] Master@127.22.30.60:35099 shutting down...
I20260808 22:50:41.260070 972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 22:50:41.260174 932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } }
I20260808 22:50:41.260710 932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 22:50:41.261432 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.261507 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
W20260808 22:50:41.261916 32187 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f -> Peer 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60:35099): Couldn't send request to peer 8a0d6b9c718243d0a13144d55681ea16. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 22:50:41.266772 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.255164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.22.30.61:37385)
0808 22:50:41.263187 (+ 8023us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.263189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.263336 (+ 147us) server_negotiation.cc:407] Connection header received
0808 22:50:41.263356 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.263356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.263376 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.263443 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.263587 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.264286 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.265698 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.266272 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.266329 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.266380 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.266680 (+ 300us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.266712 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8003}
I20260808 22:50:41.266620 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.255084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53894)
0808 22:50:41.263217 (+ 8133us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.263226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.263269 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.263490 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.263490 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.263492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.263524 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.263526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.264401 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.264402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.265598 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.265601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.266411 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.266413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.266472 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.266489 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.266532 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.266562 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8117}
I20260808 22:50:41.267514 758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 LEARNER]: Refusing update from remote peer 0a0929dc342b462a809e31b0c224cb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 22:50:41.267679 819 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260808 22:50:41.267681 675 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 22:50:41.267784 819 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260808 22:50:41.267815 819 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: stopping tablet replica
I20260808 22:50:41.268265 758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.269433 675 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: attempting to promote NON_VOTER 91f6e29cfed14feda53fb4a725048b9d to VOTER
I20260808 22:50:41.269516 798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0929dc342b462a809e31b0c224cb5f. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.269577 675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } }
I20260808 22:50:41.269610 798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: LEARNER
I20260808 22:50:41.269645 798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: NON_VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: true } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.269681 798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: LEARNER
W20260808 22:50:41.270241 32187 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f -> Peer 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60:35099): Couldn't send request to peer 8a0d6b9c718243d0a13144d55681ea16. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 22:50:41.270359 758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 LEARNER]: Refusing update from remote peer 0a0929dc342b462a809e31b0c224cb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 22:50:41.270522 675 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 22:50:41.271054 675 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.271871 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.271950 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.272001 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.272035 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.274899 758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.275928 980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.276013 980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:50:41.286778 819 master.cc:552] Master@127.22.30.60:35099 shutdown complete.
I20260808 22:50:41.286823 819 master_runner.cc:305] Clearing existing system tablet
I20260808 22:50:41.286939 819 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:41.286983 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:41.287672 819 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.288264 987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.288414 819 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.288471 819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:41.288494 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
uuid: "8a0d6b9c718243d0a13144d55681ea16"
format_stamp: "Formatted at 2026-08-08 22:50:41 on dist-test-slave-x4qh"
server_key: "53a966a4a2776be51b84468e98c1e00d"
server_key_iv: "d79d5ed60c70c843b5cd0a31fb9f853e"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.289119 819 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:41.290462 819 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 22:50:41.290531 819 log.cc:1196] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 22:50:41.290809 819 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Deleting consensus metadata
I20260808 22:50:41.290941 819 master_runner.cc:315] Copying system tablet from 127.22.30.62:37855
I20260808 22:50:41.291567 819 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: tablet copy: Beginning tablet copy session from remote peer at address 127.22.30.62:37855
I20260808 22:50:41.298484 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.291966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.22.30.62:37855)
0808 22:50:41.291996 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.291997 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.292006 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:41.292066 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.292067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.292075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.292123 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.292494 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.293269 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.294283 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.294602 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.294678 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.295778 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.295779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.295780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.295786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.297256 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.297635 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.297636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.297636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.297693 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.297953 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.297954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.297954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.298074 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.298140 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.298334 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.298406 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 22:50:41.298499 995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.291797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:48712)
0808 22:50:41.291958 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.291969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.292019 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.292130 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.292130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.292353 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.292395 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.292397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.293363 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.293364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.294162 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.294165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.294697 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.294698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.294754 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.295270 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.295275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.295685 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.297349 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.297350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.297351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.297557 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.297771 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.297772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.297772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.297840 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.298155 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.298156 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.298210 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.298296 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.298394 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260808 22:50:41.298982 32240 tablet_copy_service.cc:140] P 0a0929dc342b462a809e31b0c224cb5f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8a0d6b9c718243d0a13144d55681ea16 ({username='oryx', principal='oryx/127.22.30.60@KRBTEST.COM'} at 127.0.0.1:48712)
I20260808 22:50:41.299027 32240 tablet_copy_service.cc:161] P 0a0929dc342b462a809e31b0c224cb5f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8a0d6b9c718243d0a13144d55681ea16 at {username='oryx', principal='oryx/127.22.30.60@KRBTEST.COM'} at 127.0.0.1:48712: session id = 8a0d6b9c718243d0a13144d55681ea16-00000000000000000000000000000000
I20260808 22:50:41.299548 32240 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 22:50:41.299965 819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 22:50:41.300967 819 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: tablet copy: Starting download of 0 data blocks...
I20260808 22:50:41.301092 819 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: tablet copy: Starting download of 1 WAL segments...
I20260808 22:50:41.301798 819 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 22:50:41.302595 32240 tablet_copy_service.cc:342] P 0a0929dc342b462a809e31b0c224cb5f: Request end of tablet copy session 8a0d6b9c718243d0a13144d55681ea16-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.30.60@KRBTEST.COM'} at 127.0.0.1:48712
I20260808 22:50:41.302641 32240 tablet_copy_service.cc:433] P 0a0929dc342b462a809e31b0c224cb5f: ending tablet copy session 8a0d6b9c718243d0a13144d55681ea16-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8a0d6b9c718243d0a13144d55681ea16
I20260808 22:50:41.304919 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:41.306037 1002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.306066 997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.306398 999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.306720 819 server_base.cc:1034] running on GCE node
I20260808 22:50:41.306815 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:41.306936 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:41.308090 819 hybrid_clock.cc:648] HybridClock initialized: now 1786229441308096 us; error 44 us; skew 500 ppm
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:41.309440 819 init.cc:377] Logged in from keytab as oryx/127.22.30.60@KRBTEST.COM (short username oryx)
I20260808 22:50:41.309675 819 webserver.cc:460] Webserver started at http://127.22.30.60:44747/ using document root <none> and password file <none>
I20260808 22:50:41.309787 819 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:41.309823 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:41.310536 819 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.311054 1012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.311161 819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.311198 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
uuid: "8a0d6b9c718243d0a13144d55681ea16"
format_stamp: "Formatted at 2026-08-08 22:50:41 on dist-test-slave-x4qh"
server_key: "53a966a4a2776be51b84468e98c1e00d"
server_key_iv: "d79d5ed60c70c843b5cd0a31fb9f853e"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.311256 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.335266 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:41.336979 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:41.340029 819 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.60:35099
I20260808 22:50:41.340109 1108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.60:35099 every 8 connection(s)
I20260808 22:50:41.340348 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
I20260808 22:50:41.341253 1109 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:50:41.341718 1109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Bootstrap starting.
I20260808 22:50:41.344034 1109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:50:41.344257 1109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Bootstrap complete.
I20260808 22:50:41.344499 1109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } }
I20260808 22:50:41.344580 1109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a0d6b9c718243d0a13144d55681ea16, State: Initialized, Role: LEARNER
I20260808 22:50:41.344612 1109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } }
I20260808 22:50:41.344765 1109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 22:50:41.344952 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:41.345191 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:41.345237 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260808 22:50:41.346318 1119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.346339 1141 catalog_manager.cc:1259] Loaded cluster ID: d3226eaeff6141db90d0a38c7a02a0c7
I20260808 22:50:41.346477 1119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: LEARNER
I20260808 22:50:41.347064 1141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: loading cluster ID for follower catalog manager: success
W20260808 22:50:41.347448 966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:41.347496 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:41.347532 966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:41.348251 1141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: acquiring CA information for follower catalog manager: success
I20260808 22:50:41.348534 1141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:50:41.349347 966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:33815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=43089
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:41.349565 966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:41.349792 966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:41.355250 1147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.355237 1143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.355225 1144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.355731 966 server_base.cc:1034] running on GCE node
I20260808 22:50:41.355929 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:41.356149 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:41.357308 966 hybrid_clock.cc:648] HybridClock initialized: now 1786229441357296 us; error 26 us; skew 500 ppm
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:41.359688 966 init.cc:377] Logged in from keytab as oryx/127.22.30.1@KRBTEST.COM (short username oryx)
I20260808 22:50:41.360081 966 webserver.cc:460] Webserver started at http://127.22.30.1:43089/ using document root <none> and password file <none>
I20260808 22:50:41.360307 966 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:41.360388 966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:41.361716 966 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:41.362315 1159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.362512 966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:41.362607 966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "547b73eca41143cc8446da2fdb59e031"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "a35e7afc384fe6f3e805f4cfb7571aee"
server_key_iv: "45c151d35619242b406e6e8d26bab791"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.362924 966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.381866 966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:41.383903 966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:41.384060 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:41.384295 966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:41.384653 966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:41.384706 966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.384745 966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:41.384780 966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.391086 966 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:33815
I20260808 22:50:41.391124 1347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:33815 every 8 connection(s)
I20260808 22:50:41.391469 966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:41.394326 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 966
I20260808 22:50:41.394485 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 32528
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.60@KRBTEST.COM
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:41.403872 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.392165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.22.30.1:48591)
0808 22:50:41.392344 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.392374 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.393559 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.393811 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.393814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.394165 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.394333 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.394337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.395427 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.395428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.396401 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.396403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.396802 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.396803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.396913 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.397360 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.397368 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.399084 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.401368 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.401376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.401379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.401826 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.402084 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.402668 (+ 584us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.402669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.402917 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.403204 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.403211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.403348 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.403486 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.403634 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260808 22:50:41.404394 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.392460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:41367 (local address 127.22.30.62:37855)
0808 22:50:41.392561 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.392563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.392719 (+ 156us) server_negotiation.cc:407] Connection header received
0808 22:50:41.393676 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.393678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.393690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.393731 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.394444 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.395327 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.398015 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.398333 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.398408 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.401039 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.401041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.401041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.401050 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.402567 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.403001 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.403002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.403003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.403073 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.403282 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.403284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.403284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.403473 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.403548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.403808 (+ 260us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.403905 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 22:50:41.404891 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.392165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:41367)
0808 22:50:41.392666 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.392675 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.393582 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.393815 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.393816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.394200 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.394334 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.394340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.396888 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.396889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.397913 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.397920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.398429 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.398430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.398515 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.399010 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.399016 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.400945 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.402662 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.402663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.402663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.402884 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.403154 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.403156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.403156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.403213 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.403583 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.403584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.403639 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.404635 (+ 996us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.404751 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260808 22:50:41.405550 1352 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37855
I20260808 22:50:41.405646 1352 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.405799 1352 heartbeater.cc:511] Master 127.22.30.62:37855 requested a full tablet report, sending...
I20260808 22:50:41.406234 32209 ts_manager.cc:194] Re-registered known tserver with Master: 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815)
I20260808 22:50:41.406838 32209 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:41367
I20260808 22:50:41.408907 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.392453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:48591 (local address 127.22.30.60:35099)
0808 22:50:41.392550 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.392552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.392562 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:41.393666 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.393667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.393679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.393749 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.394490 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.395307 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.396476 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.396701 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.396743 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.399189 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.399191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.399191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.399198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.400945 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.401933 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.401934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.401935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.402011 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.402991 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.402993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.402993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.403102 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.403181 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.408661 (+ 5480us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.408773 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
I20260808 22:50:41.409247 1348 heartbeater.cc:347] Connected to a master server at 127.22.30.60:35099
I20260808 22:50:41.409315 1348 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.409400 1348 heartbeater.cc:511] Master 127.22.30.60:35099 requested a full tablet report, sending...
I20260808 22:50:41.409766 1044 ts_manager.cc:194] Registered new tserver with Master: 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815)
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
I20260808 22:50:41.413756 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:36137
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:37855
--webserver_port=37693
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:50:41.415849 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.392652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:51089 (local address 127.22.30.61:37385)
0808 22:50:41.392696 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.392698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.392882 (+ 184us) server_negotiation.cc:407] Connection header received
0808 22:50:41.393677 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.393678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.393689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.393721 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.409052 (+ 15331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.409843 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.410973 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.411188 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.411249 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.413101 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.413102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.413103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.413109 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.414689 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.415012 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.415013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.415014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.415074 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.415258 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.415260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.415260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.415443 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.415510 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.415682 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.415755 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:41.415881 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.392565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.22.30.1:51089)
0808 22:50:41.392775 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.392781 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.393582 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.393815 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.393816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.408472 (+ 14656us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.408529 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.408532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.409976 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.409976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.410893 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.410896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.411279 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.411280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.411339 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.411821 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.411826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.413019 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.414772 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.414773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.414773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.414936 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.415137 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.415138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.415138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.415184 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.415516 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.415517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.415561 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.415636 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.415753 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":58,"threads_started":1}
I20260808 22:50:41.416208 1353 heartbeater.cc:347] Connected to a master server at 127.22.30.61:37385
I20260808 22:50:41.416255 1353 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.416324 1353 heartbeater.cc:511] Master 127.22.30.61:37385 requested a full tablet report, sending...
I20260808 22:50:41.416657 736 ts_manager.cc:194] Registered new tserver with Master: 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815)
W20260808 22:50:41.490901 1361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:41.491060 1361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:41.491080 1361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:41.492053 1361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:41.492087 1361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:41.492110 1361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:41.493474 1361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:36137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=37693
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:41.493638 1361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:41.493793 1361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:41.498553 1368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.498608 1367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.498711 1361 server_base.cc:1034] running on GCE node
W20260808 22:50:41.498736 1372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.498937 1361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:41.499194 1361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:41.500331 1361 hybrid_clock.cc:648] HybridClock initialized: now 1786229441500308 us; error 52 us; skew 500 ppm
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:41.502624 1361 init.cc:377] Logged in from keytab as oryx/127.22.30.2@KRBTEST.COM (short username oryx)
I20260808 22:50:41.503075 1361 webserver.cc:460] Webserver started at http://127.22.30.2:37693/ using document root <none> and password file <none>
I20260808 22:50:41.503284 1361 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:41.503376 1361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:41.504619 1361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:41.505164 1382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.505338 1361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:41.505434 1361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "60aff0300d014752891613db853120bd"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "13f97e860e142eed2d48e1be064519f3"
server_key_iv: "dac1deeabe61a20aca940f646fbb9a17"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.505800 1361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.517181 1361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:41.519348 1361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:41.519546 1361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:41.519743 1361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:41.520016 1361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:41.520066 1361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.520102 1361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:41.520126 1361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.529732 1361 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:36137
I20260808 22:50:41.529762 1547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:36137 every 8 connection(s)
I20260808 22:50:41.530081 1361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.60@KRBTEST.COM
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:41.539299 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 1361
I20260808 22:50:41.539470 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 32762
I20260808 22:50:41.540608 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.530885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:52899 (local address 127.22.30.61:37385)
0808 22:50:41.530974 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.530976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.531182 (+ 206us) server_negotiation.cc:407] Connection header received
0808 22:50:41.532215 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.532216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.532228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.532259 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.532958 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.533668 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.534706 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.534976 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.535025 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.537557 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.537558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.537558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.537564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.539301 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.539742 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.539744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.539744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.539802 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.540022 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.540022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.540023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.540131 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.540170 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.540429 (+ 259us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.540497 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 22:50:41.541026 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.530933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:38787 (local address 127.22.30.62:37855)
0808 22:50:41.531007 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.531009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.531201 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:50:41.532215 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.532216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.532228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.532259 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.532959 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.533665 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.535092 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.535406 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.535485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.538068 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.538069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.538070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.538077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.539661 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.540041 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.540043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.540043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.540106 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.540350 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.540351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.540351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.540494 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.540542 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.540769 (+ 227us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.540877 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 22:50:41.541064 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.530724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.22.30.2:52899)
0808 22:50:41.531076 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.531112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.532095 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.532368 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.532372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.532755 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.532861 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.532863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.533762 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.533763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.534638 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.534640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.535046 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.535047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.535112 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.535517 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.535522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.537476 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.539392 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.539398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.539400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.539665 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.539860 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.539861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.539861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.539966 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.540193 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.540197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.540318 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.540675 (+ 357us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.540842 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":47,"spinlock_wait_cycles":6016,"thread_start_us":226,"threads_started":1}
I20260808 22:50:41.541390 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.530725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:38787)
0808 22:50:41.531076 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.531112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.532095 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.532372 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.532372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.532753 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.532861 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.532863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.533762 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.533763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.534638 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.534640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.535943 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.535943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.536023 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.536349 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.536353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.537996 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.539743 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.539744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.539744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.539949 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.540184 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.540185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.540186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.540252 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.540594 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.540594 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.540643 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.541112 (+ 469us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.541224 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":80,"thread_start_us":116,"threads_started":1}
I20260808 22:50:41.542223 1551 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37855
I20260808 22:50:41.542315 1551 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.542510 1551 heartbeater.cc:511] Master 127.22.30.62:37855 requested a full tablet report, sending...
I20260808 22:50:41.542532 1552 heartbeater.cc:347] Connected to a master server at 127.22.30.61:37385
I20260808 22:50:41.542562 1552 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.542603 1552 heartbeater.cc:511] Master 127.22.30.61:37385 requested a full tablet report, sending...
I20260808 22:50:41.542934 736 ts_manager.cc:194] Registered new tserver with Master: 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:41.542945 32209 ts_manager.cc:194] Re-registered known tserver with Master: 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:41.543322 32209 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:38787
I20260808 22:50:41.557490 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.530992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:59649 (local address 127.22.30.60:35099)
0808 22:50:41.531040 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.531041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.531325 (+ 284us) server_negotiation.cc:407] Connection header received
0808 22:50:41.532204 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.532205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.532221 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.532298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.532952 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.533413 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.534357 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.534545 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.534617 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.538328 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.538329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.538329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.538336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.556293 (+ 17957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.556733 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.556734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.556734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.556797 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.556972 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.556974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.556974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.557120 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.557193 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.557356 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.557435 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:50:41.557596 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.530724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.22.30.2:59649)
0808 22:50:41.531280 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.531289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.532102 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.532377 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.532378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.532674 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.532854 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.532857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.533516 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.533517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.534271 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.534273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.534624 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.534625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.534751 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.535222 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.535230 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.537996 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.556470 (+ 18474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.556471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.556472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.556661 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.556863 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.556863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.556864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.556919 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.557199 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.557200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.557244 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.557315 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.557433 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":2,"spinlock_wait_cycles":9984,"thread_start_us":112,"threads_started":1}
I20260808 22:50:41.557940 1550 heartbeater.cc:347] Connected to a master server at 127.22.30.60:35099
I20260808 22:50:41.558008 1550 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.558077 1550 heartbeater.cc:511] Master 127.22.30.60:35099 requested a full tablet report, sending...
I20260808 22:50:41.558373 1044 ts_manager.cc:194] Registered new tserver with Master: 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:41.561532 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:36781
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:37855
--webserver_port=44349
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:41.640244 1557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:41.640447 1557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:41.640484 1557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:41.641516 1557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:41.641582 1557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:41.641625 1557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:41.643072 1557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:36781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=44349
--tserver_master_addrs=127.22.30.62:37855,127.22.30.61:37385,127.22.30.60:35099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:41.643317 1557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:41.643556 1557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:41.648861 1571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.648988 1557 server_base.cc:1034] running on GCE node
W20260808 22:50:41.648871 1566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:41.649088 1565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:41.649295 1557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:41.649518 1557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:41.650655 1557 hybrid_clock.cc:648] HybridClock initialized: now 1786229441650641 us; error 32 us; skew 500 ppm
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:41.652997 1557 init.cc:377] Logged in from keytab as oryx/127.22.30.3@KRBTEST.COM (short username oryx)
I20260808 22:50:41.653410 1557 webserver.cc:460] Webserver started at http://127.22.30.3:44349/ using document root <none> and password file <none>
I20260808 22:50:41.653628 1557 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:41.653707 1557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:41.654896 1557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:41.655488 1581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.655683 1557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:50:41.655779 1557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "a3d920d87053407ab5264d8ea4ea160e"
format_stamp: "Formatted at 2026-08-08 22:50:40 on dist-test-slave-x4qh"
server_key: "00937c502f42a38c1ad2d8cd5f0e20ed"
server_key_iv: "837a2ebe11c5bab7af0bdf790c98ed43"
server_key_version: "encryptionkey@0"
I20260808 22:50:41.656065 1557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:41.674805 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.672024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.0.0.1:38336)
0808 22:50:41.672073 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.672082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.672130 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.672361 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.672362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.672364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.672424 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.672426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.673012 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.673013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.674033 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.674035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.674587 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.674588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.674654 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.674659 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.674712 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.674754 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 22:50:41.674860 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.672142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.22.30.60:35099)
0808 22:50:41.672183 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.672185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.672194 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:41.672235 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.672236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.672254 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.672335 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.672490 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.672945 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.674114 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.674502 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.674561 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.674600 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.674740 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.674782 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:41.675220 1057 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 22:50:41.675324 1119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0929dc342b462a809e31b0c224cb5f. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.675460 1119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: LEARNER
I20260808 22:50:41.676225 1557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:41.677919 1557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:41.678072 1557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:41.678277 1557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:41.678572 1557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:41.678604 1557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.678656 1557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:41.678683 1557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:41.685024 1557 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:36781
I20260808 22:50:41.685070 1749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:36781 every 8 connection(s)
I20260808 22:50:41.685372 1557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:41.686622 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 1557
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.60@KRBTEST.COM
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
Aug 08 22:50:41 dist-test-slave-x4qh krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229441, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.61@KRBTEST.COM
I20260808 22:50:41.696696 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.686504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.22.30.3:43783)
0808 22:50:41.686773 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.686780 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.687171 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.687388 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.687389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.687728 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.687980 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.687986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.688970 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.688971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.690323 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.690326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.690843 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.690844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.690965 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.691563 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.691571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.693314 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.695141 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.695147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.695149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.695453 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.695663 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.695664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.695665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.695780 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.696061 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.696067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.696209 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.696313 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.696469 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":105,"thread_start_us":69,"threads_started":1}
I20260808 22:50:41.697003 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.686672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:43783 (local address 127.22.30.60:35099)
0808 22:50:41.686719 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.686721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.686841 (+ 120us) server_negotiation.cc:407] Connection header received
0808 22:50:41.687259 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.687261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.687276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.687306 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.688096 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.688839 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.690419 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.690720 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.690786 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.693422 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.693424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.693424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.693431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.695053 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.695540 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.695541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.695541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.695595 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.695854 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.695855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.695856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.695982 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.696036 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.696836 (+ 800us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.696916 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:50:41.697261 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.686316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.22.30.3:47637)
0808 22:50:41.686495 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.686504 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.687021 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.687216 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.687217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.687686 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.687980 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.687986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.688970 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.688971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.691068 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.691070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.691550 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.691551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.691608 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.691936 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.691940 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.694383 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.696121 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.696121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.696122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.696308 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.696494 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.696495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.696495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.696543 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.696809 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.696810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.696849 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.696906 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.696983 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":1208,"thread_start_us":66,"threads_started":1}
I20260808 22:50:41.697477 1752 heartbeater.cc:347] Connected to a master server at 127.22.30.60:35099
I20260808 22:50:41.697549 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.686379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:47637 (local address 127.22.30.61:37385)
0808 22:50:41.686427 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.686429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.686552 (+ 123us) server_negotiation.cc:407] Connection header received
0808 22:50:41.687104 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.687105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.687116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.687146 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.688097 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.688834 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.691154 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.691448 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.691500 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.694487 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.694489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.694489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.694495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.696050 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.696378 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.696379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.696379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.696431 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.696602 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.696603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.696603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.696713 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.696771 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.697380 (+ 609us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.697464 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:41.697571 1752 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.697747 1752 heartbeater.cc:511] Master 127.22.30.60:35099 requested a full tablet report, sending...
I20260808 22:50:41.698293 1755 heartbeater.cc:347] Connected to a master server at 127.22.30.61:37385
I20260808 22:50:41.698344 1755 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.698457 1755 heartbeater.cc:511] Master 127.22.30.61:37385 requested a full tablet report, sending...
I20260808 22:50:41.698792 1044 ts_manager.cc:194] Registered new tserver with Master: a3d920d87053407ab5264d8ea4ea160e (127.22.30.3:36781)
I20260808 22:50:41.698793 736 ts_manager.cc:194] Registered new tserver with Master: a3d920d87053407ab5264d8ea4ea160e (127.22.30.3:36781)
I20260808 22:50:41.700657 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.688428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.22.30.61:37385)
0808 22:50:41.688723 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.688725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.688876 (+ 151us) server_negotiation.cc:407] Connection header received
0808 22:50:41.688900 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.688901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.688915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.688975 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.689917 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.690641 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.693038 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.693357 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.693418 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.695008 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.695009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.695009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.695016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.699254 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.699680 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.699682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.699682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.699743 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.699937 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.699938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.699938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.700070 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.700243 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.700454 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.700545 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":55,"threads_started":1}
I20260808 22:50:41.701251 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.688065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.22.30.60:35099)
0808 22:50:41.690740 (+ 2675us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.690741 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.690748 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:41.690768 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.690769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.690779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.690833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.692397 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.692946 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.694982 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.695516 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.695579 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.697556 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.697557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.697557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.697564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.699688 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.700054 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.700055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.700056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.700273 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.700495 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.700496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.700497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.700651 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.700737 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.701051 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.701144 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2656,"thread_start_us":66,"threads_started":1}
I20260808 22:50:41.701814 32301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.685999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:60913 (local address 127.22.30.62:37855)
0808 22:50:41.686060 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.686062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.686308 (+ 246us) server_negotiation.cc:407] Connection header received
0808 22:50:41.686996 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.686997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.687011 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.687072 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.688088 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.688823 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.690421 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.690742 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.690781 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.698028 (+ 7247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.698029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.698030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.698037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.700090 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.701045 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.701046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.701046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.701100 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.701304 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.701305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.701305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.701434 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.701475 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.701654 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.701722 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:41.701830 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.686039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:60913)
0808 22:50:41.686201 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.686230 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.686877 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.687200 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.687204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.687692 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:41.687981 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.687989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.688954 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.688955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.690317 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.690320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.690860 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.690861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.690967 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.691683 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:41.691688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:41.697915 (+ 6227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:41.700460 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.700460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.700461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.700655 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.701175 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:41.701175 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:41.701176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:41.701220 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:41.701506 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:41.701507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:41.701545 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.701629 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.701722 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":101,"thread_start_us":90,"threads_started":1}
I20260808 22:50:41.702386 1754 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37855
I20260808 22:50:41.702440 1754 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:41.702497 1754 heartbeater.cc:511] Master 127.22.30.62:37855 requested a full tablet report, sending...
I20260808 22:50:41.702704 32209 ts_manager.cc:194] Re-registered known tserver with Master: a3d920d87053407ab5264d8ea4ea160e (127.22.30.3:36781)
I20260808 22:50:41.703142 32209 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:60913
I20260808 22:50:41.704545 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.689428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.22.30.62:37855)
0808 22:50:41.689692 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.689694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.689858 (+ 164us) server_negotiation.cc:407] Connection header received
0808 22:50:41.689878 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.689879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.689889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.689940 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.691410 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.698454 (+ 7044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.699525 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.699862 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.699931 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.702009 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.702010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.702010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.702017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.703636 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.703950 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.703951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.703951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.704011 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.704159 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.704159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.704159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.704239 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.704324 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.704393 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.704451 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20260808 22:50:41.705891 32209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:41.707259 32209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:41.707933 982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: NON_VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: true } } }
I20260808 22:50:41.708039 982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.708973 982 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: attempting to promote NON_VOTER 8a0d6b9c718243d0a13144d55681ea16 to VOTER
I20260808 22:50:41.709146 982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } }
I20260808 22:50:41.709581 1057 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 LEARNER]: Refusing update from remote peer 0a0929dc342b462a809e31b0c224cb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 22:50:41.709595 758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Refusing update from remote peer 0a0929dc342b462a809e31b0c224cb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 22:50:41.709816 1783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 22:50:41.709889 675 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 22:50:41.710745 758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.710815 1057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.711903 1119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.712020 1119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:50:41.712093 980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.712182 980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:50:41.710604 1783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.713165 982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.713178 1783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.713289 982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.713297 1783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.713366 982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a0929dc342b462a809e31b0c224cb5f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:41.713409 982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:41.718413 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.714933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36781 (local address 127.0.0.1:50530)
0808 22:50:41.715489 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.715499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.715538 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.715831 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.715832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.715833 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.715873 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.715874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.716652 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.716653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.717491 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.717493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.718169 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.718170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.718231 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.718237 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.718299 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.718337 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":71,"threads_started":1}
I20260808 22:50:41.718416 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.715100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.22.30.3:36781)
0808 22:50:41.715698 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.715701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.715707 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:41.715735 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.715736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.715746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.715833 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.715932 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.716546 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.717586 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.718073 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.718168 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.718243 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.718313 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.718348 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":75,"threads_started":1}
I20260808 22:50:41.719298 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.714888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33815 (local address 127.0.0.1:43016)
0808 22:50:41.715016 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.715026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.715066 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.715438 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.715439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.715440 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.715484 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.715486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.717041 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.717042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.718151 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.718154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.719046 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.719048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.719107 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.719135 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.719198 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.719238 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1}
I20260808 22:50:41.719344 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.715076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.22.30.1:33815)
0808 22:50:41.715250 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.715254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.715265 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:50:41.715315 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.715317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.715333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.715439 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.715564 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.716943 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.718256 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.718946 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.719041 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.719123 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.719212 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.719255 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260808 22:50:41.719497 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.714994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.22.30.2:36137)
0808 22:50:41.715287 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.715290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.715441 (+ 151us) server_negotiation.cc:407] Connection header received
0808 22:50:41.715686 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.715689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.715707 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.715812 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.716022 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.717036 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.718252 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.718954 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.719039 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.719123 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.719345 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.719418 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":88,"threads_started":1}
I20260808 22:50:41.719350 1659 tablet_service.cc:1467] Processing CreateTablet for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b (DEFAULT_TABLE table=test-table [id=431fcd6fb2924fdcb591ad6e8226a663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:41.719913 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.714888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:36137 (local address 127.0.0.1:47396)
0808 22:50:41.715334 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.715342 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.715589 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.715896 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.715897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.715899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.715945 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.715947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.717115 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.717115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.718158 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.718160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.719159 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.719160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.719222 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.719226 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.719271 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.719306 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":25,"thread_start_us":83,"threads_started":1}
I20260808 22:50:41.720067 1659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:41.719992 1239 tablet_service.cc:1467] Processing CreateTablet for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b (DEFAULT_TABLE table=test-table [id=431fcd6fb2924fdcb591ad6e8226a663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:41.720290 1239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:41.720698 1452 tablet_service.cc:1467] Processing CreateTablet for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b (DEFAULT_TABLE table=test-table [id=431fcd6fb2924fdcb591ad6e8226a663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:41.721000 1452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:41.722555 1797 tablet_bootstrap.cc:492] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Bootstrap starting.
I20260808 22:50:41.723057 1797 tablet_bootstrap.cc:654] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:41.723261 1797 log.cc:824] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:41.723501 1800 tablet_bootstrap.cc:492] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Bootstrap starting.
I20260808 22:50:41.724139 1800 tablet_bootstrap.cc:654] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:41.724475 1800 log.cc:824] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:41.725036 1803 tablet_bootstrap.cc:492] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Bootstrap starting.
I20260808 22:50:41.725247 1797 tablet_bootstrap.cc:492] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: No bootstrap required, opened a new log
I20260808 22:50:41.725320 1800 tablet_bootstrap.cc:492] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: No bootstrap required, opened a new log
I20260808 22:50:41.725334 1797 ts_tablet_manager.cc:1397] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 22:50:41.725396 1800 ts_tablet_manager.cc:1397] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:41.725687 1803 tablet_bootstrap.cc:654] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:41.726329 1803 log.cc:824] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:41.727103 1803 tablet_bootstrap.cc:492] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: No bootstrap required, opened a new log
I20260808 22:50:41.727178 1803 ts_tablet_manager.cc:1397] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:41.727465 1797 raft_consensus.cc:348] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.727607 1797 raft_consensus.cc:374] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:41.727649 1797 raft_consensus.cc:729] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547b73eca41143cc8446da2fdb59e031, State: Initialized, Role: FOLLOWER
I20260808 22:50:41.727764 1797 consensus_queue.cc:260] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.728072 1797 ts_tablet_manager.cc:1428] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 22:50:41.728003 1800 raft_consensus.cc:348] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.728163 1352 heartbeater.cc:503] Master 127.22.30.62:37855 was elected leader, sending a full tablet report...
I20260808 22:50:41.728161 1800 raft_consensus.cc:374] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:41.728200 1800 raft_consensus.cc:729] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d920d87053407ab5264d8ea4ea160e, State: Initialized, Role: FOLLOWER
I20260808 22:50:41.728340 1800 consensus_queue.cc:260] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.728856 1800 ts_tablet_manager.cc:1428] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 22:50:41.729463 1754 heartbeater.cc:503] Master 127.22.30.62:37855 was elected leader, sending a full tablet report...
I20260808 22:50:41.731070 1803 raft_consensus.cc:348] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.731178 1803 raft_consensus.cc:374] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:41.731197 1803 raft_consensus.cc:729] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60aff0300d014752891613db853120bd, State: Initialized, Role: FOLLOWER
I20260808 22:50:41.731316 1803 consensus_queue.cc:260] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.731593 1803 ts_tablet_manager.cc:1428] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 22:50:41.731663 1551 heartbeater.cc:503] Master 127.22.30.62:37855 was elected leader, sending a full tablet report...
I20260808 22:50:41.749559 1809 raft_consensus.cc:482] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:41.749653 1809 raft_consensus.cc:504] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.749994 1809 leader_election.cc:288] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815), 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:41.752513 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.750183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:36137 (local address 127.22.30.3:60025)
0808 22:50:41.750219 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.750229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.750265 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.750402 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.750403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.750404 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.750462 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.750464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.751056 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.751057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.751839 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.751841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.752319 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.752320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.752391 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.752394 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.752440 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.752474 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260808 22:50:41.752707 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.750223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:60025 (local address 127.22.30.2:36137)
0808 22:50:41.750256 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.750258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.750266 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:41.750318 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.750319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.750325 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.750394 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.750538 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.750972 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.751920 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.752236 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.752319 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.752327 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.752613 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.752655 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:41.752725 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.750183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33815 (local address 127.22.30.3:52669)
0808 22:50:41.750347 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:41.750356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:41.750395 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:41.750507 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:41.750508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:41.750509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:41.750568 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.750569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.751173 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.751174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.751956 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:41.751958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.752479 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.752480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:41.752559 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.752562 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:41.752624 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:50:41.752662 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":136}
I20260808 22:50:41.752960 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.750223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:52669 (local address 127.22.30.1:33815)
0808 22:50:41.750267 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.750269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.750388 (+ 119us) server_negotiation.cc:407] Connection header received
0808 22:50:41.750442 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.750443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.750452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.750504 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:41.750668 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.751087 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.752046 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.752381 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.752474 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.752482 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.752871 (+ 389us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.752908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:41.752938 1485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb026f0ea2824ee6b1d4e7a2f03fb54b" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aff0300d014752891613db853120bd" is_pre_election: true
I20260808 22:50:41.753074 1485 raft_consensus.cc:2454] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 0.
I20260808 22:50:41.753273 1266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb026f0ea2824ee6b1d4e7a2f03fb54b" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547b73eca41143cc8446da2fdb59e031" is_pre_election: true
I20260808 22:50:41.753461 1266 raft_consensus.cc:2454] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 0.
I20260808 22:50:41.753489 1584 leader_election.cc:302] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60aff0300d014752891613db853120bd, a3d920d87053407ab5264d8ea4ea160e; no voters:
I20260808 22:50:41.753634 1809 raft_consensus.cc:2781] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:41.753715 1809 raft_consensus.cc:482] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:41.753749 1809 raft_consensus.cc:3037] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:41.754308 1809 raft_consensus.cc:504] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.754449 1809 leader_election.cc:288] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 election: Requested vote from peers 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815), 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:41.754588 1485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb026f0ea2824ee6b1d4e7a2f03fb54b" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aff0300d014752891613db853120bd"
I20260808 22:50:41.754652 1266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb026f0ea2824ee6b1d4e7a2f03fb54b" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547b73eca41143cc8446da2fdb59e031"
I20260808 22:50:41.754698 1485 raft_consensus.cc:3037] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:41.754761 1266 raft_consensus.cc:3037] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:41.755352 1266 raft_consensus.cc:2454] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 1.
I20260808 22:50:41.755365 1485 raft_consensus.cc:2454] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 1.
I20260808 22:50:41.755609 1584 leader_election.cc:302] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60aff0300d014752891613db853120bd, a3d920d87053407ab5264d8ea4ea160e; no voters:
I20260808 22:50:41.755699 1809 raft_consensus.cc:2781] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:41.755846 1809 raft_consensus.cc:686] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 1 LEADER]: Becoming Leader. State: Replica: a3d920d87053407ab5264d8ea4ea160e, State: Running, Role: LEADER
I20260808 22:50:41.755926 1809 consensus_queue.cc:237] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:41.756603 32209 catalog_manager.cc:5495] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d920d87053407ab5264d8ea4ea160e (127.22.30.3). New cstate: current_term: 1 leader_uuid: "a3d920d87053407ab5264d8ea4ea160e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:41.767041 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.764689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.22.30.3:36781)
0808 22:50:41.764726 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.764728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.764764 (+ 36us) server_negotiation.cc:407] Connection header received
0808 22:50:41.764847 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.764848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.764870 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.764945 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:41.765089 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.765563 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.766308 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.766507 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.766584 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.766748 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:41.766832 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:41.766913 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.766957 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.766975 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:41.768512 1485 raft_consensus.cc:1261] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Refusing update from remote peer a3d920d87053407ab5264d8ea4ea160e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:41.768540 1266 raft_consensus.cc:1261] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Refusing update from remote peer a3d920d87053407ab5264d8ea4ea160e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:41.768858 1809 consensus_queue.cc:1035] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:41.768999 1815 consensus_queue.cc:1035] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [LEADER]: Connected to new peer: Peer: permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:41.771162 1819 mvcc.cc:205] Tried to move back new op lower bound from 7316395793481912320 to 7316395793432875008. Current Snapshot: MvccSnapshot[applied={T|T < 7316395793481912320}]
I20260808 22:50:41.772605 1822 mvcc.cc:205] Tried to move back new op lower bound from 7316395793481912320 to 7316395793432875008. Current Snapshot: MvccSnapshot[applied={T|T < 7316395793481912320}]
I20260808 22:50:41.774547 32208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:41.774623 32208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:41.775884 32208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a0929dc342b462a809e31b0c224cb5f: Sending DeleteTablet for 3 replicas of tablet bb026f0ea2824ee6b1d4e7a2f03fb54b
I20260808 22:50:41.776129 1659 tablet_service.cc:1514] Processing DeleteTablet for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:41 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:50530
I20260808 22:50:41.776181 1239 tablet_service.cc:1514] Processing DeleteTablet for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:41 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:43016
I20260808 22:50:41.776218 1452 tablet_service.cc:1514] Processing DeleteTablet for tablet bb026f0ea2824ee6b1d4e7a2f03fb54b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:41 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47396
I20260808 22:50:41.776407 1835 tablet_replica.cc:330] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: stopping tablet replica
I20260808 22:50:41.776507 1835 raft_consensus.cc:2229] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:41.776620 1835 raft_consensus.cc:2258] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:41.776887 1837 tablet_replica.cc:330] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: stopping tablet replica
I20260808 22:50:41.777031 1837 raft_consensus.cc:2229] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:41.777172 1837 raft_consensus.cc:2258] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:41.777413 1835 ts_tablet_manager.cc:1905] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:41.778391 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 32167
I20260808 22:50:41.781558 1835 ts_tablet_manager.cc:1918] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:41.781647 1835 log.cc:1196] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/bb026f0ea2824ee6b1d4e7a2f03fb54b
I20260808 22:50:41.781991 1835 ts_tablet_manager.cc:1939] T bb026f0ea2824ee6b1d4e7a2f03fb54b P a3d920d87053407ab5264d8ea4ea160e: Deleting consensus metadata
W20260808 22:50:41.789573 1582 connection.cc:535] server connection from 127.0.0.1:50530 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:50:41.790243 1836 tablet_replica.cc:330] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: stopping tablet replica
I20260808 22:50:41.790342 1836 raft_consensus.cc:2229] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:41.790427 1836 raft_consensus.cc:2258] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:41.790843 1836 ts_tablet_manager.cc:1905] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:41.791167 1837 ts_tablet_manager.cc:1905] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:41.792353 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.777544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.22.30.60:35099)
0808 22:50:41.777591 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.777593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.777603 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:41.777635 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.777636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.777648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.777706 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.783717 (+ 6011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.784497 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.785612 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.785989 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.786052 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.788158 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.788159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.788159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.788167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.790084 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.791501 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.791502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.791503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.791563 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.791771 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.791772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.791772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.791900 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.791979 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.792186 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.792277 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 22:50:41.792755 1837 ts_tablet_manager.cc:1918] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:41.792838 1837 log.cc:1196] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/bb026f0ea2824ee6b1d4e7a2f03fb54b
I20260808 22:50:41.793100 1837 ts_tablet_manager.cc:1939] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 60aff0300d014752891613db853120bd: Deleting consensus metadata
W20260808 22:50:41.793442 1384 connection.cc:441] server connection from 127.0.0.1:47396 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47396 (request call id 1) could send its response
I20260808 22:50:41.795002 1836 ts_tablet_manager.cc:1918] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:41.795085 1836 log.cc:1196] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/bb026f0ea2824ee6b1d4e7a2f03fb54b
I20260808 22:50:41.795348 1836 ts_tablet_manager.cc:1939] T bb026f0ea2824ee6b1d4e7a2f03fb54b P 547b73eca41143cc8446da2fdb59e031: Deleting consensus metadata
W20260808 22:50:41.795707 1164 connection.cc:441] server connection from 127.0.0.1:43016 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43016 (request call id 1) could send its response
I20260808 22:50:41.801283 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.791060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.22.30.61:37385)
0808 22:50:41.793412 (+ 2352us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.793413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.793421 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:41.793438 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.793439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.793448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.793518 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.794243 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.794950 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.796038 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.796295 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.796344 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.798111 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.798112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.798112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.798119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.799914 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.800285 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.800286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.800287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.800335 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.800534 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.800535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.800535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.800662 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.800727 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.801144 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.801225 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2331}
I20260808 22:50:41.802587 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:41.791671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.22.30.60:35099)
0808 22:50:41.791708 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:41.791709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:41.795978 (+ 4269us) server_negotiation.cc:407] Connection header received
0808 22:50:41.796009 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:41.796009 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:41.796020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:41.796074 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:41.796809 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.797263 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.798325 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:41.798606 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:41.798694 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:41.800247 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:41.800249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:41.800249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:41.800255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:41.801706 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.801990 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.801991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.801991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.802048 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:41.802184 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:41.802184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:41.802185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:41.802295 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:41.802384 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:41.802435 (+ 51us) server_negotiation.cc:299] Negotiation successful
0808 22:50:41.802508 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:50:42.773007 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:42.772691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:36725)
0808 22:50:42.772829 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:42.772945 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
W20260808 22:50:42.773411 1754 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:37855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:42.773638 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:42.773405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:34163)
0808 22:50:42.773516 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:42.773589 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
I20260808 22:50:42.774045 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:42.773690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:45537)
0808 22:50:42.773927 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:42.774002 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
W20260808 22:50:42.774092 1352 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:37855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
W20260808 22:50:42.774369 1551 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:37855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:43.277141 1851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a0929dc342b462a809e31b0c224cb5f)
I20260808 22:50:43.277329 1851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } }
I20260808 22:50:43.277626 1851 leader_election.cc:288] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855), 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61:37385)
I20260808 22:50:43.277992 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.277762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49198)
0808 22:50:43.277897 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.277952 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
W20260808 22:50:43.278148 1013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:43.278618 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.278526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49220)
0808 22:50:43.278561 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.278590 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260808 22:50:43.278622 1857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a0929dc342b462a809e31b0c224cb5f)
W20260808 22:50:43.278712 1013 leader_election.cc:334] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855): Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:43.278760 1857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } }
I20260808 22:50:43.278985 1857 leader_election.cc:288] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855), 8a0d6b9c718243d0a13144d55681ea16 (127.22.30.60:35099)
I20260808 22:50:43.279423 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.279100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49222)
0808 22:50:43.279297 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.279351 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":42,"threads_started":1}
W20260808 22:50:43.279531 710 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:43.279974 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.279873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49228)
0808 22:50:43.279920 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.279950 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:50:43.280112 710 leader_election.cc:334] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855): Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:43.281433 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.277885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.22.30.61:37385)
0808 22:50:43.278012 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:43.278014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:43.278024 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:43.278136 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:43.278137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:43.278150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:43.278266 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:43.278594 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.279478 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.280788 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.281144 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.281199 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.281209 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:43.281367 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 22:50:43.281397 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260808 22:50:43.281453 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.277826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37385 (local address 127.0.0.1:53922)
0808 22:50:43.277919 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.277932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:43.278011 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:43.278318 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:43.278320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:43.278322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:43.278385 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.278387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.279567 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.279568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.280692 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.280694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.281236 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.281237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.281297 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.281301 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:43.281356 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:50:43.281391 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":52,"threads_started":1}
I20260808 22:50:43.281652 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a0d6b9c718243d0a13144d55681ea16" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91f6e29cfed14feda53fb4a725048b9d" is_pre_election: true
I20260808 22:50:43.281744 758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a0d6b9c718243d0a13144d55681ea16 in term 1.
I20260808 22:50:43.281901 1017 leader_election.cc:302] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a0d6b9c718243d0a13144d55681ea16, 91f6e29cfed14feda53fb4a725048b9d; no voters: 0a0929dc342b462a809e31b0c224cb5f
I20260808 22:50:43.282024 1851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 22:50:43.282073 1851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a0929dc342b462a809e31b0c224cb5f)
I20260808 22:50:43.282104 1851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:50:43.282611 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.279272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.22.30.60:35099)
0808 22:50:43.279426 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:43.279428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:43.279434 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:43.279464 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:43.279464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:43.279473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:43.279667 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:43.279757 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.280384 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.281843 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.282286 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.282375 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.282385 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:43.282524 (+ 139us) server_negotiation.cc:299] Negotiation successful
0808 22:50:43.282559 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260808 22:50:43.282624 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.279100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:35099 (local address 127.0.0.1:38384)
0808 22:50:43.279213 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.279222 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:43.279270 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:43.279562 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:43.279563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:43.279565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:43.279614 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.279616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.280458 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.280459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.281429 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.281432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.282381 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.282382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.282441 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.282444 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:50:43.282516 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 22:50:43.282554 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260808 22:50:43.282819 1057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f6e29cfed14feda53fb4a725048b9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8a0d6b9c718243d0a13144d55681ea16" is_pre_election: true
I20260808 22:50:43.282825 1851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } }
I20260808 22:50:43.282989 1057 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91f6e29cfed14feda53fb4a725048b9d in current term 2: Already voted for candidate 8a0d6b9c718243d0a13144d55681ea16 in this term.
I20260808 22:50:43.283015 1851 leader_election.cc:288] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [CANDIDATE]: Term 2 election: Requested vote from peers 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855), 91f6e29cfed14feda53fb4a725048b9d (127.22.30.61:37385)
I20260808 22:50:43.283154 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.283078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49236)
0808 22:50:43.283108 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.283134 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260808 22:50:43.283143 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a0d6b9c718243d0a13144d55681ea16" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91f6e29cfed14feda53fb4a725048b9d"
I20260808 22:50:43.283170 711 leader_election.cc:302] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f6e29cfed14feda53fb4a725048b9d; no voters: 0a0929dc342b462a809e31b0c224cb5f, 8a0d6b9c718243d0a13144d55681ea16
I20260808 22:50:43.283205 758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:50:43.283540 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.283442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49240)
0808 22:50:43.283499 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.283521 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260808 22:50:43.283643 1013 leader_election.cc:334] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855): Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
I20260808 22:50:43.283850 758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a0d6b9c718243d0a13144d55681ea16 in term 2.
I20260808 22:50:43.283933 1857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:50:43.284014 1017 leader_election.cc:302] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a0d6b9c718243d0a13144d55681ea16, 91f6e29cfed14feda53fb4a725048b9d; no voters: 0a0929dc342b462a809e31b0c224cb5f
I20260808 22:50:43.284088 1851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:50:43.284192 1851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [term 2 LEADER]: Becoming Leader. State: Replica: 8a0d6b9c718243d0a13144d55681ea16, State: Running, Role: LEADER
I20260808 22:50:43.284246 1851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } }
I20260808 22:50:43.284621 1863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a0d6b9c718243d0a13144d55681ea16. Latest consensus state: current_term: 2 leader_uuid: "8a0d6b9c718243d0a13144d55681ea16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:43.284695 1863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:43.284870 1865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:43.285364 1865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:43.285455 1865 catalog_manager.cc:1259] Loaded cluster ID: d3226eaeff6141db90d0a38c7a02a0c7
I20260808 22:50:43.285502 1865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:43.285689 1865 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:43.285768 1865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Loaded TSK: 0
I20260808 22:50:43.285903 1865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:50:43.334578 1040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:43.336266 1040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:43.337179 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.337085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49254)
0808 22:50:43.337132 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.337158 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 22:50:43.337206 758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [term 2 FOLLOWER]: Refusing update from remote peer 8a0d6b9c718243d0a13144d55681ea16: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 22:50:43.337453 1863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 22:50:43.337591 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.337487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.0.0.1:49268)
0808 22:50:43.337545 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.337573 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 22:50:43.337749 1013 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 -> Peer 0a0929dc342b462a809e31b0c224cb5f (127.22.30.62:37855): Couldn't send request to peer 0a0929dc342b462a809e31b0c224cb5f. Status: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 22:50:43.338359 1857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a0d6b9c718243d0a13144d55681ea16. Latest consensus state: current_term: 2 leader_uuid: "8a0d6b9c718243d0a13144d55681ea16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:43.338462 1857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:50:43.338796 1867 mvcc.cc:205] Tried to move back new op lower bound from 7316395799907250176 to 7316395799692910592. Current Snapshot: MvccSnapshot[applied={T|T < 7316395799907250176}]
I20260808 22:50:43.338908 1863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8a0d6b9c718243d0a13144d55681ea16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:43.338977 1863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:43.339051 1857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8a0d6b9c718243d0a13144d55681ea16" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0a0929dc342b462a809e31b0c224cb5f" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37855 } } peers { permanent_uuid: "91f6e29cfed14feda53fb4a725048b9d" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37385 } attrs { promote: false } } peers { permanent_uuid: "8a0d6b9c718243d0a13144d55681ea16" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 35099 } attrs { promote: false } } }
I20260808 22:50:43.339133 1857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f6e29cfed14feda53fb4a725048b9d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:50:43.339183 1868 mvcc.cc:205] Tried to move back new op lower bound from 7316395799907250176 to 7316395799692910592. Current Snapshot: MvccSnapshot[applied={T|T < 7316395799907250176 or (T in {7316395799907250176})}]
I20260808 22:50:43.344573 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.340980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33815 (local address 127.0.0.1:43024)
0808 22:50:43.341179 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.341189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:43.341227 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:43.341591 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:43.341592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:43.341595 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:43.341652 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.341654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.342385 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.342386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.343564 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.343567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.344293 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.344294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.344409 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.344414 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:43.344467 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:50:43.344509 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":39}
I20260808 22:50:43.344817 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.340909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36781 (local address 127.0.0.1:50546)
0808 22:50:43.341019 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.341029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:43.341068 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:43.341229 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:43.341230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:43.341232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:43.341292 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.341294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.342540 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.342541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.343692 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.343694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.344540 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.344540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.344612 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.344616 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:43.344671 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:50:43.344711 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":24}
I20260808 22:50:43.345016 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.340999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.22.30.3:36781)
0808 22:50:43.341090 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:43.341091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:43.341102 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:50:43.341127 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:43.341128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:43.341137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:43.341617 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:43.341660 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.342448 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.343772 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.344409 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.344539 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.344552 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:43.344822 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 22:50:43.344868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":44,"threads_started":1}
I20260808 22:50:43.345221 1659 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb38cc48bcd4065901bb549caf696fb (DEFAULT_TABLE table=test-table [id=e25b7d8b65a249d4b666def1d6feb76e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:43.345350 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.341347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.22.30.1:33815)
0808 22:50:43.341454 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:43.341456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:43.341464 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:43.341492 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:43.341492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:43.341502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:43.341714 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:43.341752 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.342283 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.343649 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.344197 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.344271 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.344294 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:43.345220 (+ 926us) server_negotiation.cc:299] Negotiation successful
0808 22:50:43.345270 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260808 22:50:43.345394 1659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb38cc48bcd4065901bb549caf696fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:43.345721 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.341163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.22.30.2:36137)
0808 22:50:43.341315 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:43.341317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:43.341525 (+ 208us) server_negotiation.cc:407] Connection header received
0808 22:50:43.341625 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:43.341626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:43.341636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:43.341675 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:43.341864 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.342570 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.344491 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.345109 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.345177 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.345190 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:43.345624 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 22:50:43.345662 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260808 22:50:43.345733 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.340996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:36137 (local address 127.0.0.1:47400)
0808 22:50:43.341466 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.341476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:43.341548 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:43.341741 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:43.341741 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:43.341743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:43.341779 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.341781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.342648 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.342649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.344403 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:43.344405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.345445 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.345446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:43.345512 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.345517 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:43.345615 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 22:50:43.345657 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":73,"threads_started":1}
I20260808 22:50:43.346167 1452 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb38cc48bcd4065901bb549caf696fb (DEFAULT_TABLE table=test-table [id=e25b7d8b65a249d4b666def1d6feb76e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:43.346307 1452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb38cc48bcd4065901bb549caf696fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:43.346853 1887 tablet_bootstrap.cc:492] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Bootstrap starting.
I20260808 22:50:43.347023 1239 tablet_service.cc:1467] Processing CreateTablet for tablet 6eb38cc48bcd4065901bb549caf696fb (DEFAULT_TABLE table=test-table [id=e25b7d8b65a249d4b666def1d6feb76e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:43.347147 1239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eb38cc48bcd4065901bb549caf696fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:43.347240 1887 tablet_bootstrap.cc:654] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:43.347869 1887 tablet_bootstrap.cc:492] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: No bootstrap required, opened a new log
I20260808 22:50:43.347930 1887 ts_tablet_manager.cc:1397] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:43.348044 1887 raft_consensus.cc:348] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.348140 1887 raft_consensus.cc:374] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:43.348160 1887 raft_consensus.cc:729] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d920d87053407ab5264d8ea4ea160e, State: Initialized, Role: FOLLOWER
I20260808 22:50:43.348201 1887 consensus_queue.cc:260] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.348337 1887 ts_tablet_manager.cc:1428] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:43.348498 1890 tablet_bootstrap.cc:492] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Bootstrap starting.
I20260808 22:50:43.348757 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.348499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:49995)
0808 22:50:43.348691 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.348726 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260808 22:50:43.348855 1890 tablet_bootstrap.cc:654] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:43.349066 1752 heartbeater.cc:503] Master 127.22.30.60:35099 was elected leader, sending a full tablet report...
I20260808 22:50:43.349311 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.349083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:59559)
0808 22:50:43.349252 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.349280 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260808 22:50:43.349454 1893 tablet_bootstrap.cc:492] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: Bootstrap starting.
I20260808 22:50:43.349752 1890 tablet_bootstrap.cc:492] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: No bootstrap required, opened a new log
I20260808 22:50:43.349820 1890 ts_tablet_manager.cc:1397] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:43.349833 1893 tablet_bootstrap.cc:654] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:43.349975 1890 raft_consensus.cc:348] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.350054 1890 raft_consensus.cc:374] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:43.350080 1890 raft_consensus.cc:729] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60aff0300d014752891613db853120bd, State: Initialized, Role: FOLLOWER
I20260808 22:50:43.350121 1890 consensus_queue.cc:260] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.350293 1890 ts_tablet_manager.cc:1428] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:43.350517 1893 tablet_bootstrap.cc:492] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: No bootstrap required, opened a new log
I20260808 22:50:43.350566 1893 ts_tablet_manager.cc:1397] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:43.350658 1550 heartbeater.cc:503] Master 127.22.30.60:35099 was elected leader, sending a full tablet report...
I20260808 22:50:43.350811 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.350502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:60317)
0808 22:50:43.350728 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.350763 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260808 22:50:43.350855 1893 raft_consensus.cc:348] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.350929 1893 raft_consensus.cc:374] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:43.350950 1893 raft_consensus.cc:729] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547b73eca41143cc8446da2fdb59e031, State: Initialized, Role: FOLLOWER
I20260808 22:50:43.351025 1893 consensus_queue.cc:260] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.351202 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.350994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:48835)
0808 22:50:43.351122 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.351164 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":76,"threads_started":1}
I20260808 22:50:43.351176 1893 ts_tablet_manager.cc:1428] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:43.351616 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.351322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:34541)
0808 22:50:43.351546 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.351578 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260808 22:50:43.351754 1348 heartbeater.cc:503] Master 127.22.30.60:35099 was elected leader, sending a full tablet report...
I20260808 22:50:43.352020 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.351921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:58549)
0808 22:50:43.351974 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.351999 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260808 22:50:43.393385 1355 tablet.cc:2369] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:43.404098 1891 raft_consensus.cc:482] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:43.404198 1891 raft_consensus.cc:504] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.404393 1891 leader_election.cc:288] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815), 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:43.404570 1266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb38cc48bcd4065901bb549caf696fb" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547b73eca41143cc8446da2fdb59e031" is_pre_election: true
I20260808 22:50:43.404654 1266 raft_consensus.cc:2454] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 0.
I20260808 22:50:43.404611 1485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb38cc48bcd4065901bb549caf696fb" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aff0300d014752891613db853120bd" is_pre_election: true
I20260808 22:50:43.404721 1485 raft_consensus.cc:2454] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 0.
I20260808 22:50:43.404824 1589 leader_election.cc:302] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547b73eca41143cc8446da2fdb59e031, a3d920d87053407ab5264d8ea4ea160e; no voters:
I20260808 22:50:43.404933 1891 raft_consensus.cc:2781] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:43.404973 1891 raft_consensus.cc:482] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:43.405001 1891 raft_consensus.cc:3037] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:43.405643 1891 raft_consensus.cc:504] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.405767 1891 leader_election.cc:288] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 election: Requested vote from peers 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815), 60aff0300d014752891613db853120bd (127.22.30.2:36137)
I20260808 22:50:43.405881 1266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb38cc48bcd4065901bb549caf696fb" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "547b73eca41143cc8446da2fdb59e031"
I20260808 22:50:43.405956 1266 raft_consensus.cc:3037] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:43.405932 1485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eb38cc48bcd4065901bb549caf696fb" candidate_uuid: "a3d920d87053407ab5264d8ea4ea160e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60aff0300d014752891613db853120bd"
I20260808 22:50:43.405988 1485 raft_consensus.cc:3037] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:43.406677 1266 raft_consensus.cc:2454] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 1.
I20260808 22:50:43.406725 1485 raft_consensus.cc:2454] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3d920d87053407ab5264d8ea4ea160e in term 1.
I20260808 22:50:43.406853 1589 leader_election.cc:302] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 547b73eca41143cc8446da2fdb59e031, a3d920d87053407ab5264d8ea4ea160e; no voters:
I20260808 22:50:43.406962 1891 raft_consensus.cc:2781] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:43.407115 1891 raft_consensus.cc:686] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 1 LEADER]: Becoming Leader. State: Replica: a3d920d87053407ab5264d8ea4ea160e, State: Running, Role: LEADER
I20260808 22:50:43.407181 1891 consensus_queue.cc:237] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } }
I20260808 22:50:43.407538 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.407422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:57121)
0808 22:50:43.407463 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.407511 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 22:50:43.407847 1040 catalog_manager.cc:5495] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d920d87053407ab5264d8ea4ea160e (127.22.30.3). New cstate: current_term: 1 leader_uuid: "a3d920d87053407ab5264d8ea4ea160e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3d920d87053407ab5264d8ea4ea160e" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:43.416495 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.414557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.22.30.3:36781)
0808 22:50:43.414604 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:43.414606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:43.414615 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:43.414639 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:43.414639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:43.414650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:43.414701 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:43.414806 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.415213 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.415957 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:43.416134 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:43.416185 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:43.416315 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:43.416339 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:43.416377 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:43.416436 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 22:50:43.416453 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 22:50:43.417266 1485 raft_consensus.cc:1261] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Refusing update from remote peer a3d920d87053407ab5264d8ea4ea160e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:43.417320 1266 raft_consensus.cc:1261] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Refusing update from remote peer a3d920d87053407ab5264d8ea4ea160e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:43.417495 1891 consensus_queue.cc:1035] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60aff0300d014752891613db853120bd" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:43.417591 1913 consensus_queue.cc:1035] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [LEADER]: Connected to new peer: Peer: permanent_uuid: "547b73eca41143cc8446da2fdb59e031" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:43.418393 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.418106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:58405)
0808 22:50:43.418329 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.418360 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
I20260808 22:50:43.418751 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.418466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:50477)
0808 22:50:43.418693 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.418724 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260808 22:50:43.419466 1921 mvcc.cc:205] Tried to move back new op lower bound from 7316395800235458560 to 7316395800196091904. Current Snapshot: MvccSnapshot[applied={T|T < 7316395800235458560}]
I20260808 22:50:43.419621 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.419508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:44641)
0808 22:50:43.419575 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.419596 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 22:50:43.420439 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.420352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.3:48907)
0808 22:50:43.420390 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.420413 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 22:50:43.420734 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.420645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.1:49513)
0808 22:50:43.420678 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.420706 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 22:50:43.421145 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.419600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37855 (local address 127.22.30.2:41419)
0808 22:50:43.421088 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.421117 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:37855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1471}
I20260808 22:50:43.421284 1040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:43.421386 1040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:43.422592 1040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a0d6b9c718243d0a13144d55681ea16: Sending DeleteTablet for 3 replicas of tablet 6eb38cc48bcd4065901bb549caf696fb
I20260808 22:50:43.422817 1659 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb38cc48bcd4065901bb549caf696fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:43 UTC) from {username='oryx', principal='oryx/127.22.30.60@KRBTEST.COM'} at 127.0.0.1:50546
I20260808 22:50:43.422935 1239 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb38cc48bcd4065901bb549caf696fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:43 UTC) from {username='oryx', principal='oryx/127.22.30.60@KRBTEST.COM'} at 127.0.0.1:43024
I20260808 22:50:43.423012 1936 tablet_replica.cc:330] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: stopping tablet replica
I20260808 22:50:43.423005 1452 tablet_service.cc:1514] Processing DeleteTablet for tablet 6eb38cc48bcd4065901bb549caf696fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:43 UTC) from {username='oryx', principal='oryx/127.22.30.60@KRBTEST.COM'} at 127.0.0.1:47400
I20260808 22:50:43.423059 1936 raft_consensus.cc:2229] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:43.423116 1936 raft_consensus.cc:2258] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:43.423115 1938 tablet_replica.cc:330] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031: stopping tablet replica
I20260808 22:50:43.423163 1938 raft_consensus.cc:2229] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:43.423173 1939 tablet_replica.cc:330] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: stopping tablet replica
I20260808 22:50:43.423200 1938 raft_consensus.cc:2258] T 6eb38cc48bcd4065901bb549caf696fb P 547b73eca41143cc8446da2fdb59e031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:43.423218 1939 raft_consensus.cc:2229] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:43.423255 1939 raft_consensus.cc:2258] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:43.423455 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 966
I20260808 22:50:43.423897 1939 ts_tablet_manager.cc:1905] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:43.425879 1939 ts_tablet_manager.cc:1918] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:43.425951 1939 log.cc:1196] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/6eb38cc48bcd4065901bb549caf696fb
I20260808 22:50:43.426182 1939 ts_tablet_manager.cc:1939] T 6eb38cc48bcd4065901bb549caf696fb P 60aff0300d014752891613db853120bd: Deleting consensus metadata
I20260808 22:50:43.426529 1014 catalog_manager.cc:4841] TS 60aff0300d014752891613db853120bd (127.22.30.2:36137): tablet 6eb38cc48bcd4065901bb549caf696fb (table test-table [id=e25b7d8b65a249d4b666def1d6feb76e]) successfully deleted
I20260808 22:50:43.431531 1936 ts_tablet_manager.cc:1905] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:43.432718 1936 ts_tablet_manager.cc:1918] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:43.432803 1936 log.cc:1196] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/6eb38cc48bcd4065901bb549caf696fb
I20260808 22:50:43.433032 1936 ts_tablet_manager.cc:1939] T 6eb38cc48bcd4065901bb549caf696fb P a3d920d87053407ab5264d8ea4ea160e: Deleting consensus metadata
I20260808 22:50:43.433351 1013 catalog_manager.cc:4841] TS a3d920d87053407ab5264d8ea4ea160e (127.22.30.3:36781): tablet 6eb38cc48bcd4065901bb549caf696fb (table test-table [id=e25b7d8b65a249d4b666def1d6feb76e]) successfully deleted
I20260808 22:50:43.435279 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:43.435166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33815 (local address 127.0.0.1:43036)
0808 22:50:43.435221 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:43.435252 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:50:43.435400 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 1361
W20260808 22:50:43.435438 1018 catalog_manager.cc:4584] TS 547b73eca41143cc8446da2fdb59e031 (127.22.30.1:33815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6eb38cc48bcd4065901bb549caf696fb: Network error: Client connection negotiation failed: client connection to 127.22.30.1:33815: connect: Connection refused (error 111)
I20260808 22:50:43.447840 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 1557
I20260808 22:50:43.460372 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 513
I20260808 22:50:43.467752 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 819
2026-08-08T22:50:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4599 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:43 dist-test-slave-x4qh krb5kdc[1943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:43 dist-test-slave-x4qh krb5kdc[1943](info): set up 2 sockets
Aug 08 22:50:43 dist-test-slave-x4qh krb5kdc[1943](info): commencing operation
krb5kdc: starting...
W20260808 22:50:44.647612 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:44 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab.
Entry for principal oryx/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab.
I20260808 22:50:44.753727 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:44891
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab
--principal=oryx/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:44.830062 1961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:44.830221 1961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:44.830240 1961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:44.831139 1961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:44.831171 1961 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 22:50:44.831182 1961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:44.831192 1961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:44.831203 1961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:44.831213 1961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:44.832441 1961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:44891
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.62.keytab
--principal=oryx/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:44.832597 1961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:44.832733 1961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:44.837037 1973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:44.837071 1968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:44.837042 1967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:44.837234 1961 server_base.cc:1034] running on GCE node
I20260808 22:50:44.837452 1961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:44.837788 1961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:44.838930 1961 hybrid_clock.cc:648] HybridClock initialized: now 1786229444838916 us; error 29 us; skew 500 ppm
Aug 08 22:50:44 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229444, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:44.841238 1961 init.cc:377] Logged in from keytab as oryx/127.22.30.62@KRBTEST.COM (short username oryx)
I20260808 22:50:44.841558 1961 webserver.cc:460] Webserver started at http://127.22.30.62:42485/ using document root <none> and password file <none>
I20260808 22:50:44.841737 1961 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:44.841773 1961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:44.841848 1961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:44.842764 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "1b7fed59515d4264b184b3fd09632859"
format_stamp: "Formatted at 2026-08-08 22:50:44 on dist-test-slave-x4qh"
server_key: "3a222567266575afb147e97e67bd6d9d"
server_key_iv: "83a9494e48c6a4c9dad535866cd56779"
server_key_version: "encryptionkey@0"
I20260808 22:50:44.843039 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "1b7fed59515d4264b184b3fd09632859"
format_stamp: "Formatted at 2026-08-08 22:50:44 on dist-test-slave-x4qh"
server_key: "3a222567266575afb147e97e67bd6d9d"
server_key_iv: "83a9494e48c6a4c9dad535866cd56779"
server_key_version: "encryptionkey@0"
I20260808 22:50:44.844246 1961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:50:44.844916 1987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:44.845093 1961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:44.845181 1961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "1b7fed59515d4264b184b3fd09632859"
format_stamp: "Formatted at 2026-08-08 22:50:44 on dist-test-slave-x4qh"
server_key: "3a222567266575afb147e97e67bd6d9d"
server_key_iv: "83a9494e48c6a4c9dad535866cd56779"
server_key_version: "encryptionkey@0"
I20260808 22:50:44.845249 1961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:44.854928 1961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:44.856500 1961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:44.856639 1961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:44.860841 1961 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:44891
I20260808 22:50:44.860889 2085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:44891 every 8 connection(s)
I20260808 22:50:44.861178 1961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:44.861773 2086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:44.863797 2086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Bootstrap starting.
I20260808 22:50:44.864420 2086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:44.864660 2086 log.cc:824] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:44.865245 2086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: No bootstrap required, opened a new log
I20260808 22:50:44.867008 2086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7fed59515d4264b184b3fd09632859" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44891 } }
I20260808 22:50:44.867132 2086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:44.867151 2086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7fed59515d4264b184b3fd09632859, State: Initialized, Role: FOLLOWER
I20260808 22:50:44.867231 2086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7fed59515d4264b184b3fd09632859" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44891 } }
I20260808 22:50:44.867307 2086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:44.867344 2086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:44.867408 2086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:44.867616 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 1961
I20260808 22:50:44.867705 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:44.867903 22648 external_mini_cluster.cc:1428] Setting key 10080f4d0c4f5f859b6dc3544d9747b7
I20260808 22:50:44.868026 2086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7fed59515d4264b184b3fd09632859" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44891 } }
I20260808 22:50:44.868122 2086 leader_election.cc:302] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b7fed59515d4264b184b3fd09632859; no voters:
I20260808 22:50:44.868330 2086 leader_election.cc:288] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:44.868350 2089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:44.868469 2089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [term 1 LEADER]: Becoming Leader. State: Replica: 1b7fed59515d4264b184b3fd09632859, State: Running, Role: LEADER
I20260808 22:50:44.868536 2089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7fed59515d4264b184b3fd09632859" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44891 } }
I20260808 22:50:44.868583 2086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:44.868873 2092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b7fed59515d4264b184b3fd09632859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7fed59515d4264b184b3fd09632859" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44891 } } }
I20260808 22:50:44.868882 2091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b7fed59515d4264b184b3fd09632859. Latest consensus state: current_term: 1 leader_uuid: "1b7fed59515d4264b184b3fd09632859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7fed59515d4264b184b3fd09632859" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44891 } } }
I20260808 22:50:44.868999 2092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:44.869009 2091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:44.869714 2098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:44.870146 2098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:44.871435 2098 catalog_manager.cc:1347] Generated new cluster ID: 3244f92669b346edad521925f2483233
I20260808 22:50:44.871497 2098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:44 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:44.878875 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:44.868376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.22.30.62:44891)
0808 22:50:44.868723 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:44.868732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:44.869226 (+ 494us) server_negotiation.cc:407] Connection header received
0808 22:50:44.869388 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:44.869395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:44.869447 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:44.869662 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:44.870435 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:44.871617 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:44.872490 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:44.873064 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:44.873166 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:44.875414 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:44.875417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:44.875418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:44.875437 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:44.877306 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:44.877650 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:44.877653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:44.877653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:44.877747 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:44.877963 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:44.877964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:44.877964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:44.878279 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:44.878425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:44.878524 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 22:50:44.878666 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":75,"threads_started":1}
I20260808 22:50:44.883584 2098 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:44.883996 2098 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:44.893886 2098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Generated new TSK 0
I20260808 22:50:44.894045 2098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab.
Entry for principal oryx/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab.
I20260808 22:50:44.927132 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44891
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:45.002264 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:45.002451 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:45.002492 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:45.003516 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:45.003549 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:45.003569 2119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:45.003587 2119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:45.005000 2119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:45.005167 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:45.005333 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:45.010193 2130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:45.010221 2127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:45.010213 2126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:45.010447 2119 server_base.cc:1034] running on GCE node
I20260808 22:50:45.010715 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:45.010952 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:45.012112 2119 hybrid_clock.cc:648] HybridClock initialized: now 1786229445012106 us; error 35 us; skew 500 ppm
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:45.014464 2119 init.cc:377] Logged in from keytab as oryx/127.22.30.1@KRBTEST.COM (short username oryx)
I20260808 22:50:45.014869 2119 webserver.cc:460] Webserver started at http://127.22.30.1:41971/ using document root <none> and password file <none>
I20260808 22:50:45.015082 2119 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:45.015151 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:45.015301 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:45.016239 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "26cf3390db8a468b9631a732249b4e90"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "6b766fdac1f93a8ad8bc9b09f4bd3813"
server_key_iv: "b16c6b2bbcaee02bd401999cf850f243"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.016613 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "26cf3390db8a468b9631a732249b4e90"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "6b766fdac1f93a8ad8bc9b09f4bd3813"
server_key_iv: "b16c6b2bbcaee02bd401999cf850f243"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.017809 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:45.018455 2140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.018613 2119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:45.018708 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "26cf3390db8a468b9631a732249b4e90"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "6b766fdac1f93a8ad8bc9b09f4bd3813"
server_key_iv: "b16c6b2bbcaee02bd401999cf850f243"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.018790 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:45.034484 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:45.036108 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:45.036263 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:45.036706 2119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:45.036768 2119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.036803 2119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:45.036829 2119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:45.043852 2119 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:39453
I20260808 22:50:45.043954 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:39453 every 8 connection(s)
I20260808 22:50:45.044240 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:45.047205 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.037190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:51005 (local address 127.22.30.62:44891)
0808 22:50:45.037263 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.037266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.037468 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:50:45.038071 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.038072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.038084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.038122 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.038989 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.039754 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.041251 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.041582 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.041651 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.044261 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.044263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.044263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.044278 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.045813 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.046214 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.046216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.046216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.046281 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.046559 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.046560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.046560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.046674 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.046748 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.047015 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.047115 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:50:45.047329 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.037111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.22.30.1:51005)
0808 22:50:45.037336 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.037380 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.037970 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.038204 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.038208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.038589 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.038884 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.038891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.039858 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.039860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.041159 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.041162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.041687 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.041689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.041842 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.042278 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.042287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.044165 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.045916 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.045923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.045925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.046135 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.046353 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.046354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.046354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.046485 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.046740 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.046747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.046881 (+ 134us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.046980 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.047126 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260808 22:50:45.047843 2323 heartbeater.cc:347] Connected to a master server at 127.22.30.62:44891
I20260808 22:50:45.047933 2323 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:45.048085 2323 heartbeater.cc:511] Master 127.22.30.62:44891 requested a full tablet report, sending...
I20260808 22:50:45.048628 2012 ts_manager.cc:194] Registered new tserver with Master: 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453)
I20260808 22:50:45.049284 2012 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:51005
I20260808 22:50:45.051656 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 2119
I20260808 22:50:45.051801 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:45.052011 22648 external_mini_cluster.cc:1428] Setting key 415c45f0ebd310a0f296b123de971239
I20260808 22:50:45.054471 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.048820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:42203 (local address 127.22.30.62:44891)
0808 22:50:45.048856 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.048857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.048923 (+ 66us) server_negotiation.cc:407] Connection header received
0808 22:50:45.048995 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.048996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.049006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.049061 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.049467 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.049923 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.050604 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.050816 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.050894 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.051849 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.051850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.051851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.051858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.053377 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.053757 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.053758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.053758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.053808 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.054000 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.054000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.054001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.054088 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.054134 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.054299 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.054357 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:45.054519 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.048733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.22.30.1:42203)
0808 22:50:45.048868 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.048877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.048938 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.049067 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.049067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.049237 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.049288 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.049290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.050027 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.050028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.050516 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.050517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.050894 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.050895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.050962 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.051326 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.051329 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.051768 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.053459 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.053461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.053461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.053685 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.053871 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.053872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.053873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.053940 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.054148 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.054149 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.054197 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.054278 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.054388 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab.
Entry for principal oryx/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab.
I20260808 22:50:45.078418 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44891
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:45.153481 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:45.153692 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:45.153730 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:45.154764 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:45.154842 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:45.154881 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:45.154938 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:45.156390 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:45.156623 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:45.156834 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:45.161643 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:45.161640 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:45.161724 2339 server_base.cc:1034] running on GCE node
W20260808 22:50:45.161679 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:45.162021 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:45.162271 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:45.163436 2339 hybrid_clock.cc:648] HybridClock initialized: now 1786229445163431 us; error 33 us; skew 500 ppm
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:45.165875 2339 init.cc:377] Logged in from keytab as oryx/127.22.30.2@KRBTEST.COM (short username oryx)
I20260808 22:50:45.166281 2339 webserver.cc:460] Webserver started at http://127.22.30.2:43287/ using document root <none> and password file <none>
I20260808 22:50:45.166504 2339 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:45.166587 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:45.166721 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:45.167742 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "57262be637be40b097420487d86dea22"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "beb3f729dba2c5e7e290ce578be5d1a0"
server_key_iv: "d9bb1d5db023256551fd26d5d72d9f0e"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.168094 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "57262be637be40b097420487d86dea22"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "beb3f729dba2c5e7e290ce578be5d1a0"
server_key_iv: "d9bb1d5db023256551fd26d5d72d9f0e"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.169559 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:45.170225 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.170387 2339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:45.170476 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "57262be637be40b097420487d86dea22"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "beb3f729dba2c5e7e290ce578be5d1a0"
server_key_iv: "d9bb1d5db023256551fd26d5d72d9f0e"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.170553 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:45.182972 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:45.185209 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:45.185372 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:45.185822 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:45.185880 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.185910 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:45.185935 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:45.193243 2339 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:43411
I20260808 22:50:45.193279 2543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:43411 every 8 connection(s)
I20260808 22:50:45.193545 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:45.195031 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.186238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57197 (local address 127.22.30.62:44891)
0808 22:50:45.186315 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.186318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.186452 (+ 134us) server_negotiation.cc:407] Connection header received
0808 22:50:45.186892 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.186893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.186907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.186984 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.187631 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.188118 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.189109 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.189431 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.189495 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.192457 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.192459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.192459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.192467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.193935 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.194422 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.194422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.194423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.194469 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.194619 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.194620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.194620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.194701 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.194740 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.194904 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.194974 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:50:45.195181 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.186195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.22.30.2:57197)
0808 22:50:45.186334 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.186368 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.186759 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.187027 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.187030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.187304 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.187538 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.187542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.188232 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.188233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.189016 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.189018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.189526 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.189528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.189682 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.190154 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.190163 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.192361 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.194016 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.194020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.194022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.194342 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.194510 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.194510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.194511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.194587 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.194732 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.194734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.194827 (+ 93us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.194892 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.195009 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260808 22:50:45.195626 2546 heartbeater.cc:347] Connected to a master server at 127.22.30.62:44891
I20260808 22:50:45.195701 2546 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:45.195863 2546 heartbeater.cc:511] Master 127.22.30.62:44891 requested a full tablet report, sending...
I20260808 22:50:45.196219 2011 ts_manager.cc:194] Registered new tserver with Master: 57262be637be40b097420487d86dea22 (127.22.30.2:43411)
I20260808 22:50:45.196859 2011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:57197
I20260808 22:50:45.201191 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.196531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.22.30.2:42009)
0808 22:50:45.196651 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.196660 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.196714 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.196833 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.196833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.197009 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.197072 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.197074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.197716 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.197717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.198144 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.198146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.198452 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.198453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.198516 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.198817 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.198820 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.199053 (+ 233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.200455 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.200455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.200456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.200603 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.200757 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.200758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.200758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.200808 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.201000 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.201001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.201031 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.201076 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.201140 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260808 22:50:45.201197 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.196620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:42009 (local address 127.22.30.62:44891)
0808 22:50:45.196662 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.196664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.196704 (+ 40us) server_negotiation.cc:407] Connection header received
0808 22:50:45.196768 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.196768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.196777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.196830 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.197148 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.197616 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.198228 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.198391 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.198439 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.199122 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.199123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.199123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.199129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.200392 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.200661 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.200662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.200662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.200706 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.200850 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.200850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.200850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.200954 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.200996 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.201089 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.201140 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 22:50:45.202781 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 2339
I20260808 22:50:45.202867 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:45.203007 22648 external_mini_cluster.cc:1428] Setting key 9499dd03f188efcdc8bae47da1cffb8a
WARNING: no policy specified for oryx/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab.
Entry for principal oryx/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab.
I20260808 22:50:45.226054 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44891
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:44851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:45.303779 2568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:45.303990 2568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:45.304028 2568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:45.305161 2568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:45.305238 2568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:45.305271 2568 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 22:50:45.305313 2568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:45.306735 2568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:44851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.30.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:45.307003 2568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:45.307243 2568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:45.312119 2580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:45.312152 2576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:45.312119 2575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:45.312355 2568 server_base.cc:1034] running on GCE node
I20260808 22:50:45.312563 2568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:45.312804 2568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:45.313937 2568 hybrid_clock.cc:648] HybridClock initialized: now 1786229445313919 us; error 34 us; skew 500 ppm
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:45.319645 2568 init.cc:377] Logged in from keytab as oryx/127.22.30.3@KRBTEST.COM (short username oryx)
I20260808 22:50:45.320092 2568 webserver.cc:460] Webserver started at http://127.22.30.3:40287/ using document root <none> and password file <none>
I20260808 22:50:45.320309 2568 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:45.320381 2568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:45.320559 2568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:45.321645 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "122d6bc36241474094a8634f32805676"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "fc45920b80f047e1ddaa79bc43daa283"
server_key_iv: "8e477cbcfe2483dd635111425ea04861"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.322003 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "122d6bc36241474094a8634f32805676"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "fc45920b80f047e1ddaa79bc43daa283"
server_key_iv: "8e477cbcfe2483dd635111425ea04861"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.323313 2568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 22:50:45.324168 2587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.324381 2568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:45.324483 2568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "122d6bc36241474094a8634f32805676"
format_stamp: "Formatted at 2026-08-08 22:50:45 on dist-test-slave-x4qh"
server_key: "fc45920b80f047e1ddaa79bc43daa283"
server_key_iv: "8e477cbcfe2483dd635111425ea04861"
server_key_version: "encryptionkey@0"
I20260808 22:50:45.324563 2568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:45.331880 2568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:45.333922 2568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:45.334072 2568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:45.334561 2568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:45.334628 2568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.334656 2568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:45.334676 2568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:45.341871 2568 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:40015
I20260808 22:50:45.342155 2568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:45.342597 2778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:40015 every 8 connection(s)
I20260808 22:50:45.343091 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.334836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36155 (local address 127.22.30.62:44891)
0808 22:50:45.334886 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.334888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.335045 (+ 157us) server_negotiation.cc:407] Connection header received
0808 22:50:45.335564 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.335566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.335579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.335631 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.336187 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.336649 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.337711 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.337902 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.337971 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.340473 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.340474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.340474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.340482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.341804 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.342222 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.342223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.342223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.342283 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.342539 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.342540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.342540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.342644 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.342725 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.342975 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.343035 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:50:45.343217 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.334821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.22.30.3:36155)
0808 22:50:45.334941 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.334969 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.335465 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.335688 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.335695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.335905 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.336075 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.336079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.336741 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.336742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.337616 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.337619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.337980 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.337981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.338093 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.338579 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.338588 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.340369 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.341891 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.341898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.341908 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.342141 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.342350 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.342351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.342351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.342483 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.342734 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.342738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.342861 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.342946 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.343049 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260808 22:50:45.343757 2780 heartbeater.cc:347] Connected to a master server at 127.22.30.62:44891
I20260808 22:50:45.343840 2780 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:45.344005 2780 heartbeater.cc:511] Master 127.22.30.62:44891 requested a full tablet report, sending...
I20260808 22:50:45.344403 2011 ts_manager.cc:194] Registered new tserver with Master: 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:45.344781 2011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:36155
I20260808 22:50:45.349887 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.344859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:39285 (local address 127.22.30.62:44891)
0808 22:50:45.344926 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.344929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.345105 (+ 176us) server_negotiation.cc:407] Connection header received
0808 22:50:45.345170 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.345171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.345180 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.345246 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.345608 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.345998 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.346577 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.346736 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.346781 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.347579 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.347580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.347581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.347586 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.349094 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.349407 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.349408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.349408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.349453 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.349588 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.349588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.349589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.349661 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.349708 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.349792 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.349841 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:50:45.349902 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.344794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.22.30.3:39285)
0808 22:50:45.345019 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.345027 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.345119 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.345253 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.345253 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.345449 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.345526 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.345528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.346078 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.346079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.346510 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.346511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.346797 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.346798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.346843 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.347189 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.347193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.347513 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.349163 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.349164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.349164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.349321 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.349506 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.349506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.349506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.349553 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.349706 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.349706 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.349736 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.349778 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.349842 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":83,"threads_started":1}
I20260808 22:50:45.350334 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 2568
I20260808 22:50:45.350492 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:45.350714 22648 external_mini_cluster.cc:1428] Setting key d66fb821aada6dcbf780539669f088a9
I20260808 22:50:45.351743 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:50:45.373028 2093 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:45.367691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.22.30.62:44891)
0808 22:50:45.367756 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.367758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.367784 (+ 26us) server_negotiation.cc:407] Connection header received
0808 22:50:45.367885 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.367886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.367895 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.367960 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.368545 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.369127 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.369921 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.370134 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.370219 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.372966 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46940: BlockingRecv error: recv got EOF from 127.0.0.1:46940 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:45.379503 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.373758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.22.30.62:44891)
0808 22:50:45.373802 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.373803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.373920 (+ 117us) server_negotiation.cc:407] Connection header received
0808 22:50:45.373973 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.373974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.373981 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.374024 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.374412 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.374766 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.375297 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.375495 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.375542 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.377362 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.377363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.377363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.377369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.378481 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.378812 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.378813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.378813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.378858 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.379052 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.379052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.379052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.379130 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.379191 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.379289 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.379342 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 22:50:45.380803 2011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:45.382182 2011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:45.388278 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.384083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39453 (local address 127.0.0.1:47626)
0808 22:50:45.384209 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.384243 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.384298 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.384836 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.384837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.384838 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.384879 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.384880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.385857 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.385858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.387035 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.387038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.387955 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.387956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.388076 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.388081 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.388141 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.388193 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":33,"thread_start_us":49,"threads_started":1}
I20260808 22:50:45.388422 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.384107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43411 (local address 127.0.0.1:46286)
0808 22:50:45.384390 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.384400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.384451 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.384789 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.384792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.384794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.384854 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.384856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.386155 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.386156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.386891 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.386895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.387959 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.387960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.388048 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.388063 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.388108 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.388166 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":18,"thread_start_us":49,"threads_started":1}
I20260808 22:50:45.388440 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.384268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:40015 (local address 127.0.0.1:56818)
0808 22:50:45.384581 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.384591 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.384648 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.384881 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.384882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.384885 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.384933 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.384935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.386130 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.386131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.387130 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.387132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.388165 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.388166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.388231 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.388235 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.388296 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.388339 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":37,"threads_started":1}
I20260808 22:50:45.388662 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.384386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.22.30.3:40015)
0808 22:50:45.384568 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.384573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.384638 (+ 65us) server_negotiation.cc:407] Connection header received
0808 22:50:45.384729 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.384731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.384747 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.384826 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.385018 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.386038 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.387213 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.388001 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.388059 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.388131 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.388533 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.388578 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260808 22:50:45.388702 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.384205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.22.30.2:43411)
0808 22:50:45.384602 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.384606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.384615 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:45.384658 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.384660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.384675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.384784 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.384965 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.386092 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.387003 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.387827 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.388244 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.388418 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.388445 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.388601 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":79,"threads_started":1}
I20260808 22:50:45.388828 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.384421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.22.30.1:39453)
0808 22:50:45.384667 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.384672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.384682 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:45.384727 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.384729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.384744 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.384959 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.385028 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.385769 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.387136 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.387854 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.387954 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.388034 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.388183 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.388224 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260808 22:50:45.389468 2667 tablet_service.cc:1467] Processing CreateTablet for tablet f342d69ee41f41e7a4d6995fdf59ded2 (DEFAULT_TABLE table=test-table [id=696c9e4e9f6242d5b835dde339e8f7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:45.389468 2224 tablet_service.cc:1467] Processing CreateTablet for tablet f342d69ee41f41e7a4d6995fdf59ded2 (DEFAULT_TABLE table=test-table [id=696c9e4e9f6242d5b835dde339e8f7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:45.389786 2667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f342d69ee41f41e7a4d6995fdf59ded2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.389781 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f342d69ee41f41e7a4d6995fdf59ded2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.392808 2809 tablet_bootstrap.cc:492] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Bootstrap starting.
I20260808 22:50:45.393110 2435 tablet_service.cc:1467] Processing CreateTablet for tablet f342d69ee41f41e7a4d6995fdf59ded2 (DEFAULT_TABLE table=test-table [id=696c9e4e9f6242d5b835dde339e8f7dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:45.393420 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f342d69ee41f41e7a4d6995fdf59ded2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.393414 2810 tablet_bootstrap.cc:492] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Bootstrap starting.
I20260808 22:50:45.393669 2809 tablet_bootstrap.cc:654] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.393999 2810 tablet_bootstrap.cc:654] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.394002 2809 log.cc:824] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:45.394327 2810 log.cc:824] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:45.394744 2809 tablet_bootstrap.cc:492] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: No bootstrap required, opened a new log
I20260808 22:50:45.394898 2809 ts_tablet_manager.cc:1397] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:50:45.395125 2810 tablet_bootstrap.cc:492] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: No bootstrap required, opened a new log
I20260808 22:50:45.395200 2810 ts_tablet_manager.cc:1397] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:45.396536 2813 tablet_bootstrap.cc:492] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Bootstrap starting.
I20260808 22:50:45.397053 2813 tablet_bootstrap.cc:654] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.397197 2810 raft_consensus.cc:348] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.397258 2809 raft_consensus.cc:348] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.397320 2813 log.cc:824] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:45.397351 2810 raft_consensus.cc:374] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.397363 2809 raft_consensus.cc:374] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.397390 2809 raft_consensus.cc:729] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d6bc36241474094a8634f32805676, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.397390 2810 raft_consensus.cc:729] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.397501 2809 consensus_queue.cc:260] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.397491 2810 consensus_queue.cc:260] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.397720 2809 ts_tablet_manager.cc:1428] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:50:45.397720 2810 ts_tablet_manager.cc:1428] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 22:50:45.397846 2780 heartbeater.cc:503] Master 127.22.30.62:44891 was elected leader, sending a full tablet report...
I20260808 22:50:45.397855 2323 heartbeater.cc:503] Master 127.22.30.62:44891 was elected leader, sending a full tablet report...
I20260808 22:50:45.398404 2813 tablet_bootstrap.cc:492] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: No bootstrap required, opened a new log
I20260808 22:50:45.398487 2813 ts_tablet_manager.cc:1397] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:45.400234 2813 raft_consensus.cc:348] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.400372 2813 raft_consensus.cc:374] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.400415 2813 raft_consensus.cc:729] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57262be637be40b097420487d86dea22, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.400521 2813 consensus_queue.cc:260] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.400776 2813 ts_tablet_manager.cc:1428] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 22:50:45.400863 2546 heartbeater.cc:503] Master 127.22.30.62:44891 was elected leader, sending a full tablet report...
W20260808 22:50:45.444082 2550 tablet.cc:2369] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:45.534379 2815 raft_consensus.cc:482] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:45.534482 2815 raft_consensus.cc:504] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.534823 2815 leader_election.cc:288] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453), 57262be637be40b097420487d86dea22 (127.22.30.2:43411)
I20260808 22:50:45.538571 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.535091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:53109 (local address 127.22.30.2:43411)
0808 22:50:45.535134 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.535136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.535147 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:50:45.535207 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.535208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.535220 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.535275 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.535493 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.536218 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.537644 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.538160 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.538252 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.538262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.538464 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.538504 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:45.538571 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.535121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:44305 (local address 127.22.30.1:39453)
0808 22:50:45.535166 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.535168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.535320 (+ 152us) server_negotiation.cc:407] Connection header received
0808 22:50:45.535349 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.535349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.535360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.535463 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.535590 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.536217 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.537643 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.538160 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.538252 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.538262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.538464 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.538503 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 22:50:45.538587 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.534999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43411 (local address 127.22.30.3:53109)
0808 22:50:45.535063 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.535074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.535137 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.535297 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.535298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.535299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.535416 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.535418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.536325 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.536326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.537531 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.537534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.538274 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.538275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.538373 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.538380 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.538442 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.538493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":91}
I20260808 22:50:45.538838 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.535000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39453 (local address 127.22.30.3:44305)
0808 22:50:45.535237 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.535246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.535297 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.535457 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.535457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.535459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.535521 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.535521 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.536325 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.536326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.537531 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.537534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.538274 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.538275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.538373 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.538380 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.538442 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.538493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":140,"thread_start_us":114,"threads_started":1}
I20260808 22:50:45.539000 2468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342d69ee41f41e7a4d6995fdf59ded2" candidate_uuid: "122d6bc36241474094a8634f32805676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57262be637be40b097420487d86dea22" is_pre_election: true
I20260808 22:50:45.539157 2468 raft_consensus.cc:2454] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122d6bc36241474094a8634f32805676 in term 0.
I20260808 22:50:45.539250 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342d69ee41f41e7a4d6995fdf59ded2" candidate_uuid: "122d6bc36241474094a8634f32805676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf3390db8a468b9631a732249b4e90" is_pre_election: true
I20260808 22:50:45.539434 2252 raft_consensus.cc:2454] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 122d6bc36241474094a8634f32805676 in term 0.
I20260808 22:50:45.539430 2594 leader_election.cc:302] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122d6bc36241474094a8634f32805676, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:45.539551 2815 raft_consensus.cc:2781] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:45.539611 2815 raft_consensus.cc:482] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:45.539633 2815 raft_consensus.cc:3037] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.540555 2815 raft_consensus.cc:504] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.540686 2815 leader_election.cc:288] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [CANDIDATE]: Term 1 election: Requested vote from peers 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453), 57262be637be40b097420487d86dea22 (127.22.30.2:43411)
I20260808 22:50:45.540827 2468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342d69ee41f41e7a4d6995fdf59ded2" candidate_uuid: "122d6bc36241474094a8634f32805676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57262be637be40b097420487d86dea22"
I20260808 22:50:45.540827 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f342d69ee41f41e7a4d6995fdf59ded2" candidate_uuid: "122d6bc36241474094a8634f32805676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf3390db8a468b9631a732249b4e90"
I20260808 22:50:45.540921 2468 raft_consensus.cc:3037] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.540917 2252 raft_consensus.cc:3037] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.541554 2468 raft_consensus.cc:2454] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122d6bc36241474094a8634f32805676 in term 1.
I20260808 22:50:45.541605 2252 raft_consensus.cc:2454] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 122d6bc36241474094a8634f32805676 in term 1.
I20260808 22:50:45.541757 2594 leader_election.cc:302] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122d6bc36241474094a8634f32805676, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:45.541847 2815 raft_consensus.cc:2781] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:45.541980 2815 raft_consensus.cc:686] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 1 LEADER]: Becoming Leader. State: Replica: 122d6bc36241474094a8634f32805676, State: Running, Role: LEADER
I20260808 22:50:45.542088 2815 consensus_queue.cc:237] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } }
I20260808 22:50:45.542883 2012 catalog_manager.cc:5495] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 122d6bc36241474094a8634f32805676 (127.22.30.3). New cstate: current_term: 1 leader_uuid: "122d6bc36241474094a8634f32805676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } health_report { overall_health: UNKNOWN } } }
W20260808 22:50:45.545282 2324 tablet.cc:2369] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:45.567222 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.565268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.22.30.3:40015)
0808 22:50:45.565319 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.565321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.565329 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:45.565392 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.565392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.565407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.565464 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:45.565582 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.566001 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.566628 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.566815 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.566858 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.566978 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:45.567052 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:45.567100 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.567132 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.567147 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:50:45.568789 2252 raft_consensus.cc:1261] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Refusing update from remote peer 122d6bc36241474094a8634f32805676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:45.568790 2468 raft_consensus.cc:1261] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Refusing update from remote peer 122d6bc36241474094a8634f32805676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:45.569114 2822 consensus_queue.cc:1035] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:45.569255 2815 consensus_queue.cc:1035] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:45.573781 2012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:45.573910 2012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:45.574621 2012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Sending DeleteTablet for 3 replicas of tablet f342d69ee41f41e7a4d6995fdf59ded2
I20260808 22:50:45.574913 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet f342d69ee41f41e7a4d6995fdf59ded2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:45 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47626
I20260808 22:50:45.574944 2667 tablet_service.cc:1514] Processing DeleteTablet for tablet f342d69ee41f41e7a4d6995fdf59ded2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:45 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56818
I20260808 22:50:45.575098 2841 tablet_replica.cc:330] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: stopping tablet replica
I20260808 22:50:45.575109 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet f342d69ee41f41e7a4d6995fdf59ded2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:45 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46286
I20260808 22:50:45.575196 2841 raft_consensus.cc:2229] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:45.575233 2012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 22:50:45.575294 2841 raft_consensus.cc:2258] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 22:50:45.575347 2012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:45.575757 2841 ts_tablet_manager.cc:1905] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:45.575963 2844 tablet_replica.cc:330] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: stopping tablet replica
I20260808 22:50:45.576092 2844 raft_consensus.cc:2229] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:45.576234 2844 raft_consensus.cc:2258] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:45.576710 2844 ts_tablet_manager.cc:1905] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:45.576877 2667 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad6ad345d14c8ead6d96e256d0ebf8 (DEFAULT_TABLE table=test-table [id=9396bc2272634084afaca1f4131dfdf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:45.576995 2667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad6ad345d14c8ead6d96e256d0ebf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.577342 2841 ts_tablet_manager.cc:1918] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:45.577415 2841 log.cc:1196] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/f342d69ee41f41e7a4d6995fdf59ded2
I20260808 22:50:45.577675 2841 ts_tablet_manager.cc:1939] T f342d69ee41f41e7a4d6995fdf59ded2 P 26cf3390db8a468b9631a732249b4e90: Deleting consensus metadata
I20260808 22:50:45.578305 2809 tablet_bootstrap.cc:492] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Bootstrap starting.
I20260808 22:50:45.578450 2435 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad6ad345d14c8ead6d96e256d0ebf8 (DEFAULT_TABLE table=test-table [id=9396bc2272634084afaca1f4131dfdf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:45.578583 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad6ad345d14c8ead6d96e256d0ebf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.578742 2809 tablet_bootstrap.cc:654] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.579006 2844 ts_tablet_manager.cc:1918] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:45.579119 2844 log.cc:1196] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/f342d69ee41f41e7a4d6995fdf59ded2
I20260808 22:50:45.579365 2809 tablet_bootstrap.cc:492] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: No bootstrap required, opened a new log
I20260808 22:50:45.579442 2809 ts_tablet_manager.cc:1397] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:45.579574 2844 ts_tablet_manager.cc:1939] T f342d69ee41f41e7a4d6995fdf59ded2 P 122d6bc36241474094a8634f32805676: Deleting consensus metadata
I20260808 22:50:45.579613 2809 raft_consensus.cc:348] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.579694 2809 raft_consensus.cc:374] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.579725 2809 raft_consensus.cc:729] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d6bc36241474094a8634f32805676, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.579773 2809 consensus_queue.cc:260] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.579876 2809 ts_tablet_manager.cc:1428] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.580196 2813 tablet_bootstrap.cc:492] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Bootstrap starting.
I20260808 22:50:45.580435 1990 catalog_manager.cc:4841] TS 122d6bc36241474094a8634f32805676 (127.22.30.3:40015): tablet f342d69ee41f41e7a4d6995fdf59ded2 (table test-table [id=696c9e4e9f6242d5b835dde339e8f7dc]) successfully deleted
I20260808 22:50:45.580647 2813 tablet_bootstrap.cc:654] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.580814 2224 tablet_service.cc:1467] Processing CreateTablet for tablet 8bad6ad345d14c8ead6d96e256d0ebf8 (DEFAULT_TABLE table=test-table [id=9396bc2272634084afaca1f4131dfdf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:45.580844 1990 catalog_manager.cc:4841] TS 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453): tablet f342d69ee41f41e7a4d6995fdf59ded2 (table test-table [id=696c9e4e9f6242d5b835dde339e8f7dc]) successfully deleted
I20260808 22:50:45.580924 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bad6ad345d14c8ead6d96e256d0ebf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.581300 2813 tablet_bootstrap.cc:492] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: No bootstrap required, opened a new log
I20260808 22:50:45.581352 2813 ts_tablet_manager.cc:1397] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:45.581487 2813 raft_consensus.cc:348] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.581560 2813 raft_consensus.cc:374] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.581585 2813 raft_consensus.cc:729] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57262be637be40b097420487d86dea22, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.581626 2813 consensus_queue.cc:260] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.581722 2813 ts_tablet_manager.cc:1428] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:45.582155 2810 tablet_bootstrap.cc:492] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Bootstrap starting.
I20260808 22:50:45.582476 2810 tablet_bootstrap.cc:654] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.583071 2810 tablet_bootstrap.cc:492] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: No bootstrap required, opened a new log
I20260808 22:50:45.583133 2810 ts_tablet_manager.cc:1397] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:45.583448 2810 raft_consensus.cc:348] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.583477 2845 tablet_replica.cc:330] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: stopping tablet replica
I20260808 22:50:45.583530 2810 raft_consensus.cc:374] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.583563 2810 raft_consensus.cc:729] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.583562 2845 raft_consensus.cc:2229] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:45.583604 2810 consensus_queue.cc:260] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.583649 2845 raft_consensus.cc:2258] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:45.583704 2810 ts_tablet_manager.cc:1428] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:45.584138 2845 ts_tablet_manager.cc:1905] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:45.585620 2845 ts_tablet_manager.cc:1918] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:45.585693 2845 log.cc:1196] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/f342d69ee41f41e7a4d6995fdf59ded2
I20260808 22:50:45.585935 2845 ts_tablet_manager.cc:1939] T f342d69ee41f41e7a4d6995fdf59ded2 P 57262be637be40b097420487d86dea22: Deleting consensus metadata
I20260808 22:50:45.586304 1993 catalog_manager.cc:4841] TS 57262be637be40b097420487d86dea22 (127.22.30.2:43411): tablet f342d69ee41f41e7a4d6995fdf59ded2 (table test-table [id=696c9e4e9f6242d5b835dde339e8f7dc]) successfully deleted
W20260808 22:50:45.592695 2782 tablet.cc:2369] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:50:45.611486 2819 raft_consensus.cc:482] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:45.611544 2819 raft_consensus.cc:504] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.611824 2819 leader_election.cc:288] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:45.614701 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.612028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:48733 (local address 127.22.30.1:39453)
0808 22:50:45.612129 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.612131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.612140 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:45.612166 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.612167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.612175 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.612238 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.612437 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.612896 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.613848 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.614314 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.614383 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.614392 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.614610 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.614645 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 22:50:45.614715 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.611961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39453 (local address 127.22.30.2:48733)
0808 22:50:45.612000 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.612011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.612053 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.612258 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.612259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.612261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.612347 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.612349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.612982 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.612983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.613737 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.613739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.614402 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.614403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.614496 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.614501 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.614598 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.614647 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260808 22:50:45.614985 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad6ad345d14c8ead6d96e256d0ebf8" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf3390db8a468b9631a732249b4e90" is_pre_election: true
I20260808 22:50:45.615063 2252 raft_consensus.cc:2454] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 0.
I20260808 22:50:45.615221 2362 leader_election.cc:302] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cf3390db8a468b9631a732249b4e90, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:45.615345 2819 raft_consensus.cc:2781] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:45.615399 2819 raft_consensus.cc:482] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:45.615427 2819 raft_consensus.cc:3037] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.615803 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.612170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:35153 (local address 127.22.30.3:40015)
0808 22:50:45.612213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.612216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.612259 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:50:45.612338 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.612340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.612348 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.612400 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.613176 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.613829 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.614894 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.615415 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.615460 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.615469 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.615710 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.615742 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:45.615816 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.612063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:40015 (local address 127.22.30.2:35153)
0808 22:50:45.612193 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.612203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.612281 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.613005 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.613006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.613008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.613099 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.613100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.613901 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.613902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.614772 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.614775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.615522 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.615522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.615592 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.615594 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.615682 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.615738 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260808 22:50:45.616082 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad6ad345d14c8ead6d96e256d0ebf8" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676" is_pre_election: true
I20260808 22:50:45.616195 2709 raft_consensus.cc:2454] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 0.
I20260808 22:50:45.616194 2819 raft_consensus.cc:504] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.616343 2819 leader_election.cc:288] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 election: Requested vote from peers 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:45.616475 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad6ad345d14c8ead6d96e256d0ebf8" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf3390db8a468b9631a732249b4e90"
I20260808 22:50:45.616534 2252 raft_consensus.cc:3037] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.616547 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bad6ad345d14c8ead6d96e256d0ebf8" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676"
I20260808 22:50:45.616618 2709 raft_consensus.cc:3037] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.617493 2252 raft_consensus.cc:2454] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 1.
I20260808 22:50:45.617645 2709 raft_consensus.cc:2454] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 1.
I20260808 22:50:45.617673 2362 leader_election.cc:302] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cf3390db8a468b9631a732249b4e90, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:45.617749 2819 raft_consensus.cc:2781] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:45.617838 2819 raft_consensus.cc:686] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 1 LEADER]: Becoming Leader. State: Replica: 57262be637be40b097420487d86dea22, State: Running, Role: LEADER
I20260808 22:50:45.617899 2819 consensus_queue.cc:237] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.618553 2012 catalog_manager.cc:5495] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57262be637be40b097420487d86dea22 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "57262be637be40b097420487d86dea22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } health_report { overall_health: UNKNOWN } } }
Aug 08 22:50:45 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229444, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.62@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:45.658789 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.651898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44891 (local address 127.0.0.1:46960)
0808 22:50:45.652059 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.652066 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.652114 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.652232 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.652233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.652483 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:45.652521 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.652523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.653370 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.653371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.653830 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.653831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.654264 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.654264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.654302 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.654805 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:45.654810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:45.656195 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:45.657851 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.657857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.657859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.658024 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.658219 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:45.658219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:45.658220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:45.658288 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:45.658542 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:45.658548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:45.658584 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.658633 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.658713 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260808 22:50:45.658874 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.651988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.22.30.62:44891)
0808 22:50:45.652057 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.652059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.652122 (+ 63us) server_negotiation.cc:407] Connection header received
0808 22:50:45.652147 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.652148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.652158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.652223 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:45.652612 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.653297 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.653922 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.654186 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.654261 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.656296 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:45.656298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:45.656298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:45.656305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:45.657773 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.658094 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.658095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.658096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.658155 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:45.658357 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:45.658358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:45.658358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:45.658479 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:45.658547 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.658678 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.658777 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":47}
I20260808 22:50:45.661113 2012 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46960:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 22:50:45.661384 2012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:45.662771 2435 tablet_service.cc:1467] Processing CreateTablet for tablet e9425f67e6b2491abc400b3a7c80613a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe468f5f16ef491196fb7c5389e46e5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:45.662796 2667 tablet_service.cc:1467] Processing CreateTablet for tablet e9425f67e6b2491abc400b3a7c80613a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe468f5f16ef491196fb7c5389e46e5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:45.662765 2224 tablet_service.cc:1467] Processing CreateTablet for tablet e9425f67e6b2491abc400b3a7c80613a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe468f5f16ef491196fb7c5389e46e5e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 22:50:45.662920 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9425f67e6b2491abc400b3a7c80613a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.662920 2667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9425f67e6b2491abc400b3a7c80613a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.662920 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9425f67e6b2491abc400b3a7c80613a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:45.664570 2810 tablet_bootstrap.cc:492] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: Bootstrap starting.
I20260808 22:50:45.664570 2809 tablet_bootstrap.cc:492] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: Bootstrap starting.
I20260808 22:50:45.664980 2813 tablet_bootstrap.cc:492] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: Bootstrap starting.
I20260808 22:50:45.665004 2809 tablet_bootstrap.cc:654] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.665004 2810 tablet_bootstrap.cc:654] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.665510 2813 tablet_bootstrap.cc:654] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:45.665738 2809 tablet_bootstrap.cc:492] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: No bootstrap required, opened a new log
I20260808 22:50:45.665791 2809 ts_tablet_manager.cc:1397] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:45.665937 2809 raft_consensus.cc:348] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.665997 2809 raft_consensus.cc:374] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.665997 2810 tablet_bootstrap.cc:492] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: No bootstrap required, opened a new log
I20260808 22:50:45.666018 2809 raft_consensus.cc:729] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d6bc36241474094a8634f32805676, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.666033 2810 ts_tablet_manager.cc:1397] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:45.666057 2809 consensus_queue.cc:260] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.666167 2809 ts_tablet_manager.cc:1428] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:45.666183 2810 raft_consensus.cc:348] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.666254 2810 raft_consensus.cc:374] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.666292 2810 raft_consensus.cc:729] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.666270 2815 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:45.666337 2815 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:45.666342 2810 consensus_queue.cc:260] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.666432 2810 ts_tablet_manager.cc:1428] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:45.666587 2816 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:45.666676 2816 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:45.667176 2813 tablet_bootstrap.cc:492] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: No bootstrap required, opened a new log
I20260808 22:50:45.667228 2813 ts_tablet_manager.cc:1397] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:45.667459 2813 raft_consensus.cc:348] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.667534 2813 raft_consensus.cc:374] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:45.667562 2813 raft_consensus.cc:729] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57262be637be40b097420487d86dea22, State: Initialized, Role: FOLLOWER
I20260808 22:50:45.667598 2813 consensus_queue.cc:260] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.667678 2813 ts_tablet_manager.cc:1428] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:45.667747 2819 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:45.667832 2819 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:45.990294 2816 raft_consensus.cc:482] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:45.990447 2816 raft_consensus.cc:504] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.990835 2816 leader_election.cc:288] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57262be637be40b097420487d86dea22 (127.22.30.2:43411), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:45.995105 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.991143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60699 (local address 127.22.30.2:43411)
0808 22:50:45.991192 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.991195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.991346 (+ 151us) server_negotiation.cc:407] Connection header received
0808 22:50:45.991422 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.991424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.991433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.991491 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.991744 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.992486 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.994126 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.994676 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.994748 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.994758 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.994998 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.995040 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 22:50:45.995105 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.991162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:58721 (local address 127.22.30.3:40015)
0808 22:50:45.991214 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:45.991215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:45.991346 (+ 131us) server_negotiation.cc:407] Connection header received
0808 22:50:45.991422 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:45.991424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:45.991437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:45.991491 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:50:45.991744 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.992492 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.994127 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.994676 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.994748 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.994759 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:45.994990 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 22:50:45.995036 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:50:45.995101 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.991056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43411 (local address 127.22.30.1:60699)
0808 22:50:45.991243 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.991254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.991331 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.991515 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.991516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.991537 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.991668 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.991669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.992574 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.992575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.994022 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.994025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.994789 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.994789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.994868 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.994875 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.994983 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.995011 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":165,"thread_start_us":108,"threads_started":1}
I20260808 22:50:45.995101 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:45.991056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:40015 (local address 127.22.30.1:58721)
0808 22:50:45.991243 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:45.991254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:45.991331 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:45.991515 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:45.991516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:45.991518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:50:45.991665 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.991667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.992575 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.992575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.994022 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:45.994025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:45.994771 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:45.994772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:45.994868 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:45.994877 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 22:50:45.994937 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:50:45.994989 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":42,"thread_start_us":111,"threads_started":1}
I20260808 22:50:45.995420 2468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9425f67e6b2491abc400b3a7c80613a" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57262be637be40b097420487d86dea22" is_pre_election: true
I20260808 22:50:45.995507 2468 raft_consensus.cc:2454] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 0.
I20260808 22:50:45.995664 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9425f67e6b2491abc400b3a7c80613a" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676" is_pre_election: true
I20260808 22:50:45.995802 2709 raft_consensus.cc:2454] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 0.
I20260808 22:50:45.995786 2146 leader_election.cc:302] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cf3390db8a468b9631a732249b4e90, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:45.995930 2816 raft_consensus.cc:2781] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:45.995971 2816 raft_consensus.cc:482] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:45.995987 2816 raft_consensus.cc:3037] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.996731 2816 raft_consensus.cc:504] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.996881 2816 leader_election.cc:288] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 election: Requested vote from peers 57262be637be40b097420487d86dea22 (127.22.30.2:43411), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:45.997025 2468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9425f67e6b2491abc400b3a7c80613a" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57262be637be40b097420487d86dea22"
I20260808 22:50:45.997133 2468 raft_consensus.cc:3037] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.997104 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9425f67e6b2491abc400b3a7c80613a" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676"
I20260808 22:50:45.997224 2709 raft_consensus.cc:3037] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:45.997730 2468 raft_consensus.cc:2454] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 1.
I20260808 22:50:45.997910 2709 raft_consensus.cc:2454] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 1.
I20260808 22:50:45.997913 2146 leader_election.cc:302] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cf3390db8a468b9631a732249b4e90, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:45.998023 2816 raft_consensus.cc:2781] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:45.998157 2816 raft_consensus.cc:686] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [term 1 LEADER]: Becoming Leader. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Running, Role: LEADER
I20260808 22:50:45.998238 2816 consensus_queue.cc:237] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:45.998628 2878 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: TxnStatusTablet state changed. Reason: New leader 26cf3390db8a468b9631a732249b4e90. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:45.998725 2878 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:45.998940 2880 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 22:50:45.999042 2880 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 22:50:45.998992 2012 catalog_manager.cc:5495] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26cf3390db8a468b9631a732249b4e90 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:46.019055 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.016488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39453 (local address 127.0.0.1:47640)
0808 22:50:46.016564 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:46.016576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:46.016630 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:46.016787 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:46.016788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:46.016789 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:46.016831 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:46.016832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.017613 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.017614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:46.018090 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:46.018091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.018583 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.018584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:46.018637 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.018651 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.018918 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.018925 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 22:50:46.018962 (+ 37us) client_negotiation.cc:240] Negotiation successful
0808 22:50:46.019000 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 22:50:46.019076 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.016576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.22.30.1:39453)
0808 22:50:46.016620 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.016623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.016633 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:46.016691 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.016692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.016709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.016776 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:46.016876 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.017550 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.018193 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.018500 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.018576 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.018740 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.018853 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.018927 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.018966 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.018984 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:50:46.020183 2468 raft_consensus.cc:1261] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Refusing update from remote peer 26cf3390db8a468b9631a732249b4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.020251 2709 raft_consensus.cc:1261] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Refusing update from remote peer 26cf3390db8a468b9631a732249b4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.020417 2878 consensus_queue.cc:1035] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.020521 2816 consensus_queue.cc:1035] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.021076 2819 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: TxnStatusTablet state changed. Reason: New leader 26cf3390db8a468b9631a732249b4e90. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:46.021162 2819 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:46.021227 2815 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: TxnStatusTablet state changed. Reason: New leader 26cf3390db8a468b9631a732249b4e90. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:46.021303 2815 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:46.022022 2816 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:46.022096 2816 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:46.022614 2829 mvcc.cc:205] Tried to move back new op lower bound from 7316395810896592896 to 7316395810809364480. Current Snapshot: MvccSnapshot[applied={T|T < 7316395810896592896 or (T in {7316395810896592896})}]
I20260808 22:50:46.022797 2819 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:46.022873 2819 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 57262be637be40b097420487d86dea22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:46.025985 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.023571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.22.30.2:43411)
0808 22:50:46.023605 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.023607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.023616 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:46.023651 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.023651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.023667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.023743 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:46.023822 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.024547 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.025214 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.025514 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.025584 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.025695 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.025801 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.025864 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.025899 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.025916 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:46.031158 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.028775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:59083 (local address 127.22.30.1:39453)
0808 22:50:46.028811 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.028812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.028970 (+ 158us) server_negotiation.cc:407] Connection header received
0808 22:50:46.029044 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.029044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.029051 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.029095 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:46.029228 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.029639 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.030600 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.030752 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.030797 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.030993 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.031013 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.031047 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.031107 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.031121 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:50:46.031222 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.028707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39453 (local address 127.22.30.2:59083)
0808 22:50:46.028902 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:46.028913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:46.028983 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:46.029110 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:46.029111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:46.029112 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:46.029170 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:46.029171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.029712 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.029713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:46.030504 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:46.030506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.030836 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.030838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:46.030904 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.030919 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.031051 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.031053 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:50:46.031106 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:50:46.031144 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260808 22:50:46.032214 2815 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:46.032315 2815 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 122d6bc36241474094a8634f32805676: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 22:50:46.032222 2827 mvcc.cc:205] Tried to move back new op lower bound from 7316395810896592896 to 7316395810809364480. Current Snapshot: MvccSnapshot[applied={T|T < 7316395810896592896}]
I20260808 22:50:46.032989 2878 tablet_replica.cc:439] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } } }
I20260808 22:50:46.033085 2878 tablet_replica.cc:442] T e9425f67e6b2491abc400b3a7c80613a P 26cf3390db8a468b9631a732249b4e90: This TxnStatusTablet replica's current role is: LEADER
I20260808 22:50:46.033923 2252 raft_consensus.cc:1261] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Refusing update from remote peer 57262be637be40b097420487d86dea22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.033932 2707 raft_consensus.cc:1261] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Refusing update from remote peer 57262be637be40b097420487d86dea22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.034399 2853 consensus_queue.cc:1035] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.034554 2819 consensus_queue.cc:1035] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.036190 2831 mvcc.cc:205] Tried to move back new op lower bound from 7316395810953220096 to 7316395809251377152. Current Snapshot: MvccSnapshot[applied={T|T < 7316395810953220096}]
I20260808 22:50:46.042502 2901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.039873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43411 (local address 127.22.30.1:37609)
0808 22:50:46.040019 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:46.040029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:46.040081 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:46.040215 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:46.040215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:46.040216 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 22:50:46.040258 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:46.040259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.040978 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.040979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:46.041717 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:46.041719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.042165 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.042165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:46.042220 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.042232 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.042390 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.042391 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 22:50:46.042419 (+ 28us) client_negotiation.cc:240] Negotiation successful
0808 22:50:46.042445 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260808 22:50:46.042500 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.040060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:37609 (local address 127.22.30.2:43411)
0808 22:50:46.040097 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.040099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.040125 (+ 26us) server_negotiation.cc:407] Connection header received
0808 22:50:46.040145 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.040146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.040155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.040214 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:46.040302 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.040907 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.041825 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.042085 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.042150 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.042302 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.042331 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.042392 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.042424 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.042442 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:50:46.054847 2012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:46.054937 2012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:46.055634 2012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Sending DeleteTablet for 3 replicas of tablet 8bad6ad345d14c8ead6d96e256d0ebf8
I20260808 22:50:46.055943 2667 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad6ad345d14c8ead6d96e256d0ebf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56818
I20260808 22:50:46.056042 2844 tablet_replica.cc:330] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: stopping tablet replica
I20260808 22:50:46.056097 2844 raft_consensus.cc:2229] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:46.056139 2844 raft_consensus.cc:2258] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.056428 2844 ts_tablet_manager.cc:1905] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.058068 2844 ts_tablet_manager.cc:1918] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:46.058130 2844 log.cc:1196] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/8bad6ad345d14c8ead6d96e256d0ebf8
I20260808 22:50:46.058373 2844 ts_tablet_manager.cc:1939] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 122d6bc36241474094a8634f32805676: Deleting consensus metadata
I20260808 22:50:46.059491 1990 catalog_manager.cc:4841] TS 122d6bc36241474094a8634f32805676 (127.22.30.3:40015): tablet 8bad6ad345d14c8ead6d96e256d0ebf8 (table test-table [id=9396bc2272634084afaca1f4131dfdf2]) successfully deleted
I20260808 22:50:46.060631 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad6ad345d14c8ead6d96e256d0ebf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47626
I20260808 22:50:46.060631 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bad6ad345d14c8ead6d96e256d0ebf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46286
I20260808 22:50:46.060730 2845 tablet_replica.cc:330] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: stopping tablet replica
I20260808 22:50:46.060731 2841 tablet_replica.cc:330] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: stopping tablet replica
I20260808 22:50:46.060793 2841 raft_consensus.cc:2229] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:46.060793 2845 raft_consensus.cc:2229] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:46.060853 2841 raft_consensus.cc:2258] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.060899 2845 raft_consensus.cc:2258] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.061203 2845 ts_tablet_manager.cc:1905] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.061208 2841 ts_tablet_manager.cc:1905] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.062857 2841 ts_tablet_manager.cc:1918] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:46.062857 2845 ts_tablet_manager.cc:1918] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:46.062913 2841 log.cc:1196] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/8bad6ad345d14c8ead6d96e256d0ebf8
I20260808 22:50:46.062913 2845 log.cc:1196] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/8bad6ad345d14c8ead6d96e256d0ebf8
I20260808 22:50:46.063149 2845 ts_tablet_manager.cc:1939] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 57262be637be40b097420487d86dea22: Deleting consensus metadata
I20260808 22:50:46.063154 2841 ts_tablet_manager.cc:1939] T 8bad6ad345d14c8ead6d96e256d0ebf8 P 26cf3390db8a468b9631a732249b4e90: Deleting consensus metadata
I20260808 22:50:46.063495 1993 catalog_manager.cc:4841] TS 57262be637be40b097420487d86dea22 (127.22.30.2:43411): tablet 8bad6ad345d14c8ead6d96e256d0ebf8 (table test-table [id=9396bc2272634084afaca1f4131dfdf2]) successfully deleted
I20260808 22:50:46.063490 1990 catalog_manager.cc:4841] TS 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453): tablet 8bad6ad345d14c8ead6d96e256d0ebf8 (table test-table [id=9396bc2272634084afaca1f4131dfdf2]) successfully deleted
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[1943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:50:46.089493 2093 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:46.082675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.22.30.62:44891)
0808 22:50:46.082712 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.082714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.082860 (+ 146us) server_negotiation.cc:407] Connection header received
0808 22:50:46.082896 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.082897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.082909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.082971 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:46.083664 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.084348 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.085494 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.085798 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.085884 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.089433 (+ 3549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46966: BlockingRecv error: recv got EOF from 127.0.0.1:46966 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[1943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM
I20260808 22:50:46.098879 2093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.090212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.22.30.62:44891)
0808 22:50:46.090248 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.090250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.090501 (+ 251us) server_negotiation.cc:407] Connection header received
0808 22:50:46.090530 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.090531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.090541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.090629 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:46.091196 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.091807 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.092991 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.093299 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.093375 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.095992 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:46.095993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:46.095994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:46.096000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:46.097642 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:46.098026 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:46.098027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:46.098027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:46.098084 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:46.098318 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:46.098319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:46.098319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:46.098443 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:46.098536 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.098682 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.098773 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:50:46.100540 2012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:46.100710 2012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:46.102025 2224 tablet_service.cc:1467] Processing CreateTablet for tablet 14b62efb3b024660ae77c5b5596121a6 (DEFAULT_TABLE table=test-table [id=7eaa9545991d4264901ec7b657c86649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:46.102113 2667 tablet_service.cc:1467] Processing CreateTablet for tablet 14b62efb3b024660ae77c5b5596121a6 (DEFAULT_TABLE table=test-table [id=7eaa9545991d4264901ec7b657c86649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:46.102203 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b62efb3b024660ae77c5b5596121a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:46.102249 2667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b62efb3b024660ae77c5b5596121a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:46.103665 2435 tablet_service.cc:1467] Processing CreateTablet for tablet 14b62efb3b024660ae77c5b5596121a6 (DEFAULT_TABLE table=test-table [id=7eaa9545991d4264901ec7b657c86649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:46.103775 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b62efb3b024660ae77c5b5596121a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:46.103847 2810 tablet_bootstrap.cc:492] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Bootstrap starting.
I20260808 22:50:46.104205 2810 tablet_bootstrap.cc:654] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:46.104879 2810 tablet_bootstrap.cc:492] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: No bootstrap required, opened a new log
I20260808 22:50:46.104923 2810 ts_tablet_manager.cc:1397] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:46.105078 2810 raft_consensus.cc:348] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.105146 2810 raft_consensus.cc:374] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:46.105165 2810 raft_consensus.cc:729] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Initialized, Role: FOLLOWER
I20260808 22:50:46.105162 2809 tablet_bootstrap.cc:492] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Bootstrap starting.
I20260808 22:50:46.105216 2810 consensus_queue.cc:260] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.105305 2810 ts_tablet_manager.cc:1428] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:46.105506 2809 tablet_bootstrap.cc:654] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:46.105798 2813 tablet_bootstrap.cc:492] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Bootstrap starting.
I20260808 22:50:46.106204 2813 tablet_bootstrap.cc:654] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:46.107579 2809 tablet_bootstrap.cc:492] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: No bootstrap required, opened a new log
I20260808 22:50:46.107643 2809 ts_tablet_manager.cc:1397] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:46.107816 2809 raft_consensus.cc:348] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.107883 2809 raft_consensus.cc:374] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:46.107903 2809 raft_consensus.cc:729] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d6bc36241474094a8634f32805676, State: Initialized, Role: FOLLOWER
I20260808 22:50:46.107939 2809 consensus_queue.cc:260] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.108028 2809 ts_tablet_manager.cc:1428] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:46.108405 2813 tablet_bootstrap.cc:492] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: No bootstrap required, opened a new log
I20260808 22:50:46.108454 2813 ts_tablet_manager.cc:1397] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 22:50:46.108664 2813 raft_consensus.cc:348] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.108745 2813 raft_consensus.cc:374] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:46.108767 2813 raft_consensus.cc:729] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57262be637be40b097420487d86dea22, State: Initialized, Role: FOLLOWER
I20260808 22:50:46.108807 2813 consensus_queue.cc:260] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.108894 2813 ts_tablet_manager.cc:1428] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:46.263868 2878 raft_consensus.cc:482] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:46.263954 2878 raft_consensus.cc:504] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.264160 2878 leader_election.cc:288] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57262be637be40b097420487d86dea22 (127.22.30.2:43411), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:46.264403 2468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b62efb3b024660ae77c5b5596121a6" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57262be637be40b097420487d86dea22" is_pre_election: true
I20260808 22:50:46.264412 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b62efb3b024660ae77c5b5596121a6" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676" is_pre_election: true
I20260808 22:50:46.264514 2468 raft_consensus.cc:2454] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 0.
I20260808 22:50:46.264511 2709 raft_consensus.cc:2454] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 0.
I20260808 22:50:46.264678 2141 leader_election.cc:302] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122d6bc36241474094a8634f32805676, 26cf3390db8a468b9631a732249b4e90; no voters:
I20260808 22:50:46.264775 2878 raft_consensus.cc:2781] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:46.264817 2878 raft_consensus.cc:482] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:46.264851 2878 raft_consensus.cc:3037] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:46.265656 2878 raft_consensus.cc:504] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.265801 2878 leader_election.cc:288] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 election: Requested vote from peers 57262be637be40b097420487d86dea22 (127.22.30.2:43411), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:46.265962 2468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b62efb3b024660ae77c5b5596121a6" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57262be637be40b097420487d86dea22"
I20260808 22:50:46.266031 2468 raft_consensus.cc:3037] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:46.266594 2468 raft_consensus.cc:2454] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 1.
I20260808 22:50:46.266562 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b62efb3b024660ae77c5b5596121a6" candidate_uuid: "26cf3390db8a468b9631a732249b4e90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676"
I20260808 22:50:46.266657 2709 raft_consensus.cc:3037] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:46.266775 2146 leader_election.cc:302] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cf3390db8a468b9631a732249b4e90, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:46.266887 2878 raft_consensus.cc:2781] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:46.266944 2878 raft_consensus.cc:686] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 1 LEADER]: Becoming Leader. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Running, Role: LEADER
I20260808 22:50:46.266992 2878 consensus_queue.cc:237] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.267571 2709 raft_consensus.cc:2454] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26cf3390db8a468b9631a732249b4e90 in term 1.
I20260808 22:50:46.267561 2012 catalog_manager.cc:5495] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26cf3390db8a468b9631a732249b4e90 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "26cf3390db8a468b9631a732249b4e90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:46.285892 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.283938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.22.30.1:39453)
0808 22:50:46.283989 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.283991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.283997 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:50:46.284035 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.284035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.284043 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.284089 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:46.284205 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.284684 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.285361 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.285542 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.285601 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.285721 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.285743 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.285777 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.285830 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.285846 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:50:46.286621 2709 raft_consensus.cc:1261] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Refusing update from remote peer 26cf3390db8a468b9631a732249b4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.286648 2468 raft_consensus.cc:1261] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Refusing update from remote peer 26cf3390db8a468b9631a732249b4e90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.286832 2878 consensus_queue.cc:1035] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.286928 2816 consensus_queue.cc:1035] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.289955 2011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:46.290056 2011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:46.290666 2011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Sending DeleteTablet for 3 replicas of tablet 14b62efb3b024660ae77c5b5596121a6
I20260808 22:50:46.290928 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b62efb3b024660ae77c5b5596121a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46286
I20260808 22:50:46.290956 2667 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b62efb3b024660ae77c5b5596121a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56818
I20260808 22:50:46.291007 2845 tablet_replica.cc:330] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: stopping tablet replica
I20260808 22:50:46.290997 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b62efb3b024660ae77c5b5596121a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47626
I20260808 22:50:46.291036 2844 tablet_replica.cc:330] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: stopping tablet replica
I20260808 22:50:46.291049 2845 raft_consensus.cc:2229] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:46.291066 2841 tablet_replica.cc:330] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: stopping tablet replica
I20260808 22:50:46.291081 2844 raft_consensus.cc:2229] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:46.291086 2845 raft_consensus.cc:2258] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.291105 2841 raft_consensus.cc:2229] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:46.291117 2844 raft_consensus.cc:2258] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.291180 2841 raft_consensus.cc:2258] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.291445 2844 ts_tablet_manager.cc:1905] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.291456 2841 ts_tablet_manager.cc:1905] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.291661 2845 ts_tablet_manager.cc:1905] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.292917 2841 ts_tablet_manager.cc:1918] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:46.292964 2841 log.cc:1196] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/14b62efb3b024660ae77c5b5596121a6
I20260808 22:50:46.292889 2011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:50:46.293023 2011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:50:46.293038 2845 ts_tablet_manager.cc:1918] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:46.293095 2845 log.cc:1196] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/14b62efb3b024660ae77c5b5596121a6
I20260808 22:50:46.293186 2841 ts_tablet_manager.cc:1939] T 14b62efb3b024660ae77c5b5596121a6 P 26cf3390db8a468b9631a732249b4e90: Deleting consensus metadata
I20260808 22:50:46.293310 2845 ts_tablet_manager.cc:1939] T 14b62efb3b024660ae77c5b5596121a6 P 57262be637be40b097420487d86dea22: Deleting consensus metadata
I20260808 22:50:46.293401 2844 ts_tablet_manager.cc:1918] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:50:46.293474 2844 log.cc:1196] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/14b62efb3b024660ae77c5b5596121a6
I20260808 22:50:46.293640 2844 ts_tablet_manager.cc:1939] T 14b62efb3b024660ae77c5b5596121a6 P 122d6bc36241474094a8634f32805676: Deleting consensus metadata
I20260808 22:50:46.293967 1990 catalog_manager.cc:4841] TS 122d6bc36241474094a8634f32805676 (127.22.30.3:40015): tablet 14b62efb3b024660ae77c5b5596121a6 (table test-table [id=7eaa9545991d4264901ec7b657c86649]) successfully deleted
I20260808 22:50:46.294049 1990 catalog_manager.cc:4841] TS 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453): tablet 14b62efb3b024660ae77c5b5596121a6 (table test-table [id=7eaa9545991d4264901ec7b657c86649]) successfully deleted
I20260808 22:50:46.294543 1993 catalog_manager.cc:4841] TS 57262be637be40b097420487d86dea22 (127.22.30.2:43411): tablet 14b62efb3b024660ae77c5b5596121a6 (table test-table [id=7eaa9545991d4264901ec7b657c86649]) successfully deleted
I20260808 22:50:46.294852 2224 tablet_service.cc:1467] Processing CreateTablet for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 (DEFAULT_TABLE table=test-table [id=d8b1a9850e6c44619a91416eb30aa20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:46.294896 2435 tablet_service.cc:1467] Processing CreateTablet for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 (DEFAULT_TABLE table=test-table [id=d8b1a9850e6c44619a91416eb30aa20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:46.294962 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:46.295001 2435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:46.295893 2810 tablet_bootstrap.cc:492] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90: Bootstrap starting.
I20260808 22:50:46.296116 2667 tablet_service.cc:1467] Processing CreateTablet for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 (DEFAULT_TABLE table=test-table [id=d8b1a9850e6c44619a91416eb30aa20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:50:46.296141 2813 tablet_bootstrap.cc:492] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Bootstrap starting.
I20260808 22:50:46.296207 2667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:46.296229 2810 tablet_bootstrap.cc:654] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:46.296419 2813 tablet_bootstrap.cc:654] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:46.297019 2813 tablet_bootstrap.cc:492] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: No bootstrap required, opened a new log
I20260808 22:50:46.297063 2813 ts_tablet_manager.cc:1397] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:46.297153 2813 raft_consensus.cc:348] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.297209 2813 raft_consensus.cc:374] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:46.297227 2813 raft_consensus.cc:729] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57262be637be40b097420487d86dea22, State: Initialized, Role: FOLLOWER
I20260808 22:50:46.297263 2813 consensus_queue.cc:260] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.297262 2809 tablet_bootstrap.cc:492] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Bootstrap starting.
I20260808 22:50:46.297340 2813 ts_tablet_manager.cc:1428] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:46.297552 2809 tablet_bootstrap.cc:654] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:46.297683 2810 tablet_bootstrap.cc:492] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90: No bootstrap required, opened a new log
I20260808 22:50:46.297725 2810 ts_tablet_manager.cc:1397] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:50:46.297837 2810 raft_consensus.cc:348] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.297897 2810 raft_consensus.cc:374] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:46.297910 2810 raft_consensus.cc:729] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cf3390db8a468b9631a732249b4e90, State: Initialized, Role: FOLLOWER
I20260808 22:50:46.297936 2810 consensus_queue.cc:260] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.298027 2810 ts_tablet_manager.cc:1428] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:46.298434 2809 tablet_bootstrap.cc:492] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: No bootstrap required, opened a new log
I20260808 22:50:46.298486 2809 ts_tablet_manager.cc:1397] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:46.298594 2809 raft_consensus.cc:348] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.298661 2809 raft_consensus.cc:374] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:46.298691 2809 raft_consensus.cc:729] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d6bc36241474094a8634f32805676, State: Initialized, Role: FOLLOWER
I20260808 22:50:46.298731 2809 consensus_queue.cc:260] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.298833 2809 ts_tablet_manager.cc:1428] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:46.497195 2912 raft_consensus.cc:482] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:50:46.497282 2912 raft_consensus.cc:504] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.497473 2912 leader_election.cc:288] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:46.497689 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c391291b0ff40f29f3ff4e49a1ca7d5" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf3390db8a468b9631a732249b4e90" is_pre_election: true
I20260808 22:50:46.497725 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c391291b0ff40f29f3ff4e49a1ca7d5" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676" is_pre_election: true
I20260808 22:50:46.497808 2252 raft_consensus.cc:2454] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 0.
I20260808 22:50:46.497816 2709 raft_consensus.cc:2454] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 0.
I20260808 22:50:46.497988 2362 leader_election.cc:302] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cf3390db8a468b9631a732249b4e90, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:46.498101 2912 raft_consensus.cc:2781] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:50:46.498147 2912 raft_consensus.cc:482] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:50:46.498167 2912 raft_consensus.cc:3037] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:46.498924 2912 raft_consensus.cc:504] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.499049 2912 leader_election.cc:288] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 election: Requested vote from peers 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453), 122d6bc36241474094a8634f32805676 (127.22.30.3:40015)
I20260808 22:50:46.499219 2709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c391291b0ff40f29f3ff4e49a1ca7d5" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d6bc36241474094a8634f32805676"
I20260808 22:50:46.499222 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c391291b0ff40f29f3ff4e49a1ca7d5" candidate_uuid: "57262be637be40b097420487d86dea22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cf3390db8a468b9631a732249b4e90"
I20260808 22:50:46.499297 2252 raft_consensus.cc:3037] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:46.499297 2709 raft_consensus.cc:3037] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:46.500067 2709 raft_consensus.cc:2454] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 1.
I20260808 22:50:46.500068 2252 raft_consensus.cc:2454] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57262be637be40b097420487d86dea22 in term 1.
I20260808 22:50:46.500243 2362 leader_election.cc:302] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122d6bc36241474094a8634f32805676, 57262be637be40b097420487d86dea22; no voters:
I20260808 22:50:46.500335 2912 raft_consensus.cc:2781] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:46.500386 2912 raft_consensus.cc:686] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 1 LEADER]: Becoming Leader. State: Replica: 57262be637be40b097420487d86dea22, State: Running, Role: LEADER
I20260808 22:50:46.500432 2912 consensus_queue.cc:237] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } }
I20260808 22:50:46.500998 2012 catalog_manager.cc:5495] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57262be637be40b097420487d86dea22 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "57262be637be40b097420487d86dea22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57262be637be40b097420487d86dea22" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 } health_report { overall_health: UNKNOWN } } }
I20260808 22:50:46.528220 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.525966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.22.30.2:43411)
0808 22:50:46.526027 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:46.526029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:46.526058 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:50:46.526131 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:46.526132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:46.526140 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:46.526191 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:50:46.526306 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.526767 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.527548 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:46.527730 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:46.527813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:46.527955 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:50:46.528004 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:50:46.528075 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:46.528129 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 22:50:46.528150 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:50:46.530077 2252 raft_consensus.cc:1261] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Refusing update from remote peer 57262be637be40b097420487d86dea22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.530084 2709 raft_consensus.cc:1261] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Refusing update from remote peer 57262be637be40b097420487d86dea22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:50:46.530396 2853 consensus_queue.cc:1035] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122d6bc36241474094a8634f32805676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.530498 2912 consensus_queue.cc:1035] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cf3390db8a468b9631a732249b4e90" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:50:46.540140 2011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:46.540208 2011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:50:46.540753 2011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7fed59515d4264b184b3fd09632859: Sending DeleteTablet for 3 replicas of tablet 6c391291b0ff40f29f3ff4e49a1ca7d5
I20260808 22:50:46.540973 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:47626
I20260808 22:50:46.540987 2667 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56818
I20260808 22:50:46.540971 2435 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:50:46 UTC) from {username='oryx', principal='oryx/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46286
I20260808 22:50:46.541065 2844 tablet_replica.cc:330] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: stopping tablet replica
I20260808 22:50:46.541111 2844 raft_consensus.cc:2229] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:46.541141 2844 raft_consensus.cc:2258] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.541215 2841 tablet_replica.cc:330] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90: stopping tablet replica
I20260808 22:50:46.541268 2841 raft_consensus.cc:2229] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:50:46.541297 2841 raft_consensus.cc:2258] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 26cf3390db8a468b9631a732249b4e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.541344 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 2119
I20260808 22:50:46.541361 2845 tablet_replica.cc:330] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: stopping tablet replica
I20260808 22:50:46.541407 2845 raft_consensus.cc:2229] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:50:46.541426 2844 ts_tablet_manager.cc:1905] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.541461 2845 raft_consensus.cc:2258] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:50:46.549098 2845 ts_tablet_manager.cc:1905] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:50:46.551088 2845 ts_tablet_manager.cc:1918] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:46.551158 2845 log.cc:1196] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/6c391291b0ff40f29f3ff4e49a1ca7d5
I20260808 22:50:46.551443 2845 ts_tablet_manager.cc:1939] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 57262be637be40b097420487d86dea22: Deleting consensus metadata
I20260808 22:50:46.551740 1993 catalog_manager.cc:4841] TS 57262be637be40b097420487d86dea22 (127.22.30.2:43411): tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 (table test-table [id=d8b1a9850e6c44619a91416eb30aa20a]) successfully deleted
I20260808 22:50:46.555662 2844 ts_tablet_manager.cc:1918] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 22:50:46.555724 2844 log.cc:1196] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/6c391291b0ff40f29f3ff4e49a1ca7d5
I20260808 22:50:46.555927 2844 ts_tablet_manager.cc:1939] T 6c391291b0ff40f29f3ff4e49a1ca7d5 P 122d6bc36241474094a8634f32805676: Deleting consensus metadata
I20260808 22:50:46.556206 1990 catalog_manager.cc:4841] TS 122d6bc36241474094a8634f32805676 (127.22.30.3:40015): tablet 6c391291b0ff40f29f3ff4e49a1ca7d5 (table test-table [id=d8b1a9850e6c44619a91416eb30aa20a]) successfully deleted
W20260808 22:50:46.558204 1990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:39453 (error 108)
I20260808 22:50:46.558549 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 2339
I20260808 22:50:46.558739 2962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:46.558548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39453 (local address 127.0.0.1:47662)
0808 22:50:46.558646 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:46.558697 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:39453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":35,"threads_started":1}
W20260808 22:50:46.566148 1990 catalog_manager.cc:4584] TS 26cf3390db8a468b9631a732249b4e90 (127.22.30.1:39453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c391291b0ff40f29f3ff4e49a1ca7d5: Network error: Client connection negotiation failed: client connection to 127.22.30.1:39453: connect: Connection refused (error 111)
I20260808 22:50:46.570621 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 2568
I20260808 22:50:46.577099 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 1961
2026-08-08T22:50:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3111 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[2965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[2965](info): set up 2 sockets
Aug 08 22:50:46 dist-test-slave-x4qh krb5kdc[2965](info): commencing operation
krb5kdc: starting...
W20260808 22:50:47.763348 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:47 dist-test-slave-x4qh krb5kdc[2965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:47.874548 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39221
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:39221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:47.949625 2983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:47.949795 2983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:47.949813 2983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:47.950703 2983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:47.950731 2983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:47.950743 2983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:47.950754 2983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:47.950764 2983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:47.951924 2983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:39221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:47.952075 2983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:47.952208 2983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:47.956849 2991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:47.956899 2983 server_base.cc:1034] running on GCE node
W20260808 22:50:47.956851 2992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:47.956907 2995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:47.957320 2983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:47.957589 2983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:47.958752 2983 hybrid_clock.cc:648] HybridClock initialized: now 1786229447958747 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T22:50:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1377 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:50:48 dist-test-slave-x4qh krb5kdc[3005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:50:48 dist-test-slave-x4qh krb5kdc[3005](info): set up 2 sockets
Aug 08 22:50:48 dist-test-slave-x4qh krb5kdc[3005](info): commencing operation
krb5kdc: starting...
W20260808 22:50:49.124919 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:50:49.234210 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38483
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:49.313329 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:49.313519 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:49.313540 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:49.314569 3020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:49.314610 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:49.314623 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:49.314635 3020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:49.314687 3020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:49.316079 3020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:49.316326 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:49.316553 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:49.321271 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:49.321296 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:49.321281 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:49.321393 3020 server_base.cc:1034] running on GCE node
I20260808 22:50:49.321696 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:49.321967 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:49.323112 3020 hybrid_clock.cc:648] HybridClock initialized: now 1786229449323100 us; error 26 us; skew 500 ppm
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:49.325516 3020 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:49.325958 3020 webserver.cc:460] Webserver started at http://127.22.30.62:34101/ using document root <none> and password file <none>
I20260808 22:50:49.326189 3020 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:49.326256 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:49.326406 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:49.327472 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "1bccc25095374548bf430d612ef55393"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "e47d0de05fe429ce667abdd45fc8886b"
server_key_iv: "1e65ddd2086193d92a7f52a607e1b7e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.327813 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "1bccc25095374548bf430d612ef55393"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "e47d0de05fe429ce667abdd45fc8886b"
server_key_iv: "1e65ddd2086193d92a7f52a607e1b7e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.329093 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:50:49.329810 3043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.329977 3020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:49.330072 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "1bccc25095374548bf430d612ef55393"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "e47d0de05fe429ce667abdd45fc8886b"
server_key_iv: "1e65ddd2086193d92a7f52a607e1b7e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.330139 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:49.346547 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:49.348341 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:49.348508 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:49.352581 3020 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:38483
I20260808 22:50:49.352632 3136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:38483 every 8 connection(s)
I20260808 22:50:49.352954 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:49.353734 3137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:50:49.355891 3137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Bootstrap starting.
I20260808 22:50:49.356389 3137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Neither blocks nor log segments found. Creating new log.
I20260808 22:50:49.356622 3137 log.cc:824] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:49.357203 3137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: No bootstrap required, opened a new log
I20260808 22:50:49.358687 3137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:49.358822 3137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:50:49.358845 3137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bccc25095374548bf430d612ef55393, State: Initialized, Role: FOLLOWER
I20260808 22:50:49.359004 3137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:49.359103 3137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:49.359088 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 3020
I20260808 22:50:49.359146 3137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:49.359200 3137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:50:49.359216 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:50:49.359459 22648 external_mini_cluster.cc:1428] Setting key ce5727ca75ce03e44c5097fe75e2a241
I20260808 22:50:49.360013 3137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:49.360184 3137 leader_election.cc:302] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bccc25095374548bf430d612ef55393; no voters:
I20260808 22:50:49.360340 3137 leader_election.cc:288] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:50:49.360532 3146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:50:49.360584 3137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:49.360720 3146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 LEADER]: Becoming Leader. State: Replica: 1bccc25095374548bf430d612ef55393, State: Running, Role: LEADER
I20260808 22:50:49.360858 3146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:49.361258 3147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bccc25095374548bf430d612ef55393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } } }
I20260808 22:50:49.361364 3147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:49.361265 3150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bccc25095374548bf430d612ef55393. Latest consensus state: current_term: 1 leader_uuid: "1bccc25095374548bf430d612ef55393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } } }
I20260808 22:50:49.361591 3160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:49.362078 3160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:49.361595 3150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:49.363559 3160 catalog_manager.cc:1347] Generated new cluster ID: 480aecf0eb044a7092ed046949fcedb3
I20260808 22:50:49.363615 3160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:49.371847 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.359955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.22.30.62:38483)
0808 22:50:49.360177 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:49.360186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:49.360219 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:50:49.360345 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:49.360352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:49.360412 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:49.361436 (+ 1024us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:49.361541 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.363140 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.364199 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.364855 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.364985 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.367537 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:49.367541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:49.367541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:49.367552 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:49.370178 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.370563 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.370566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.370567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.370675 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.370894 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.370895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.370895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.371231 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:49.371430 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:49.371501 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 22:50:49.371638 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260808 22:50:49.394223 3160 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:50:49.394923 3160 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:49.400573 3160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Generated new TSK 0
I20260808 22:50:49.400741 3160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:50:49.431567 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38483
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:49.513377 3177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:49.513554 3177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:49.513576 3177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:49.514609 3177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:49.514640 3177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:49.514663 3177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:49.516113 3177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:49.516291 3177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:49.516450 3177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:49.521405 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:49.521406 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:49.521420 3177 server_base.cc:1034] running on GCE node
W20260808 22:50:49.521422 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:49.521857 3177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:49.522092 3177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:49.523231 3177 hybrid_clock.cc:648] HybridClock initialized: now 1786229449523210 us; error 34 us; skew 500 ppm
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:49.525687 3177 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:50:49.526141 3177 webserver.cc:460] Webserver started at http://127.22.30.1:35459/ using document root <none> and password file <none>
I20260808 22:50:49.526399 3177 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:49.526443 3177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:49.526593 3177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:49.527631 3177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "f2ac83c3110d4e55a25b789b54b32665"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0bf0bb51c2c611665f46ce6ebc3db44c"
server_key_iv: "f187db98d145c9901329b718ad6c3376"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.528015 3177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "f2ac83c3110d4e55a25b789b54b32665"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0bf0bb51c2c611665f46ce6ebc3db44c"
server_key_iv: "f187db98d145c9901329b718ad6c3376"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.529311 3177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:49.530036 3198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.530224 3177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:49.530320 3177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "f2ac83c3110d4e55a25b789b54b32665"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0bf0bb51c2c611665f46ce6ebc3db44c"
server_key_iv: "f187db98d145c9901329b718ad6c3376"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.530392 3177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:49.542474 3177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:49.544237 3177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:49.544415 3177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:49.544624 3177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:49.544922 3177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:49.544983 3177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.545023 3177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:49.545053 3177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.550782 3177 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:44573
I20260808 22:50:49.550882 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:44573 every 8 connection(s)
I20260808 22:50:49.551167 3177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:50:49.556370 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 3177
I20260808 22:50:49.556480 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:50:49.556610 22648 external_mini_cluster.cc:1428] Setting key 21da917be8ec3b4c756ce44496179e66
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:49.560086 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.551906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:40205 (local address 127.22.30.62:38483)
0808 22:50:49.551965 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:49.551968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:49.552142 (+ 174us) server_negotiation.cc:407] Connection header received
0808 22:50:49.552585 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:49.552586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:49.552597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:49.552641 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:49.553521 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.553995 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.554988 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.555208 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.555306 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.557456 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:49.557458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:49.557458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:49.557464 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:49.558692 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.559095 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.559097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.559097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.559171 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.559454 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.559455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.559455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.559571 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:49.559646 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:49.559924 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 22:50:49.560006 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:50:49.560220 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.551850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:40205)
0808 22:50:49.552022 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:49.552062 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:49.552483 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:49.552709 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:49.552712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:49.553242 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:49.553421 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:49.553425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.554089 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.554090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:49.554885 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:49.554888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.555320 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.555321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:49.555473 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.555978 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:49.555985 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:49.557353 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:49.558787 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:49.558793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:49.558795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:49.559000 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:49.559249 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:49.559250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:49.559250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:49.559355 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:49.559649 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:49.559656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:49.559798 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 22:50:49.559890 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 22:50:49.560026 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260808 22:50:49.560636 3370 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38483
I20260808 22:50:49.560724 3370 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:49.560876 3370 heartbeater.cc:511] Master 127.22.30.62:38483 requested a full tablet report, sending...
I20260808 22:50:49.561431 3078 ts_manager.cc:194] Registered new tserver with Master: f2ac83c3110d4e55a25b789b54b32665 (127.22.30.1:44573)
I20260808 22:50:49.562327 3078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:40205
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:50:49.582125 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38483
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:49.666949 3379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:49.667114 3379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:49.667135 3379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:49.668263 3379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:49.668308 3379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:49.668362 3379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:49.669703 3379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:49.669971 3379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:49.670188 3379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:49.674890 3392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:49.674970 3385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:49.674890 3386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:49.675086 3379 server_base.cc:1034] running on GCE node
I20260808 22:50:49.675278 3379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:49.675526 3379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:49.676692 3379 hybrid_clock.cc:648] HybridClock initialized: now 1786229449676686 us; error 38 us; skew 500 ppm
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:49.679028 3379 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:50:49.679488 3379 webserver.cc:460] Webserver started at http://127.22.30.2:40415/ using document root <none> and password file <none>
I20260808 22:50:49.679725 3379 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:49.679805 3379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:49.679968 3379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:49.680869 3379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "9d180a496b9e441d8a7a74f28db4d810"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0a826ceb124e98dfb800e3f33af1a6c6"
server_key_iv: "e008941bd5aa609f3ba7d42f5b57c1e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.681267 3379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "9d180a496b9e441d8a7a74f28db4d810"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0a826ceb124e98dfb800e3f33af1a6c6"
server_key_iv: "e008941bd5aa609f3ba7d42f5b57c1e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.682603 3379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:50:49.683342 3405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.683539 3379 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:49.683634 3379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "9d180a496b9e441d8a7a74f28db4d810"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0a826ceb124e98dfb800e3f33af1a6c6"
server_key_iv: "e008941bd5aa609f3ba7d42f5b57c1e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.683720 3379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:49.705577 3379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:49.707149 3379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:49.707309 3379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:49.707562 3379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:49.707866 3379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:49.707919 3379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.707949 3379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:49.707973 3379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.713570 3379 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:45301
I20260808 22:50:49.713622 3578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:45301 every 8 connection(s)
I20260808 22:50:49.713917 3379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:49.717291 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 3379
I20260808 22:50:49.717389 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:50:49.717607 22648 external_mini_cluster.cc:1428] Setting key 20a846c13864b2f5922ac9d910db8cec
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:49.723188 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.714541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:52557 (local address 127.22.30.62:38483)
0808 22:50:49.714621 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:49.714623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:49.714832 (+ 209us) server_negotiation.cc:407] Connection header received
0808 22:50:49.715222 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:49.715223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:49.715233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:49.715290 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:49.716117 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.716611 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.717740 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.718054 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.718099 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.720553 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:49.720554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:49.720554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:49.720561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:49.721804 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.722257 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.722258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.722258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.722319 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.722595 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.722595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.722595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.722725 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:49.722794 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:49.723075 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 22:50:49.723134 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:50:49.723476 3585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.714519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:52557)
0808 22:50:49.714707 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:49.714739 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:49.715146 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:49.715338 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:49.715340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:49.715880 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:49.716043 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:49.716048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.716703 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.716704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:49.717611 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:49.717614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.718186 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.718187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:49.718347 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.718897 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:49.718908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:49.720465 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:49.721883 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:49.721890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:49.721891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:49.722188 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:49.722389 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:49.722389 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:49.722390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:49.722529 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:49.722797 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:49.722804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:49.722949 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 22:50:49.723056 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 22:50:49.723201 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
I20260808 22:50:49.723968 3579 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38483
I20260808 22:50:49.724071 3579 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:49.724261 3579 heartbeater.cc:511] Master 127.22.30.62:38483 requested a full tablet report, sending...
I20260808 22:50:49.724642 3078 ts_manager.cc:194] Registered new tserver with Master: 9d180a496b9e441d8a7a74f28db4d810 (127.22.30.2:45301)
I20260808 22:50:49.725020 3078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:52557
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:50:49.742142 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38483
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:49.819531 3591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:49.819706 3591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:49.819727 3591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:49.820792 3591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:49.820827 3591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:49.820852 3591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:49.822163 3591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:49.822464 3591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:49.822649 3591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:49.827625 3600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:49.827664 3591 server_base.cc:1034] running on GCE node
W20260808 22:50:49.827633 3603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:49.827632 3610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:49.828037 3591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:49.828308 3591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:49.829478 3591 hybrid_clock.cc:648] HybridClock initialized: now 1786229449829474 us; error 37 us; skew 500 ppm
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:49.831936 3591 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:50:49.832438 3591 webserver.cc:460] Webserver started at http://127.22.30.3:44097/ using document root <none> and password file <none>
I20260808 22:50:49.832733 3591 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:49.832808 3591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:49.832943 3591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:50:49.834075 3591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "6981006fabd644e39266e0cd90b3df50"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "640b81ee0f95c25c47f3532597056384"
server_key_iv: "c93b2e93f38bc4388ae22b298cbd6c5b"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.834434 3591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "6981006fabd644e39266e0cd90b3df50"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "640b81ee0f95c25c47f3532597056384"
server_key_iv: "c93b2e93f38bc4388ae22b298cbd6c5b"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.835759 3591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:50:49.836472 3619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.836632 3591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:50:49.836721 3591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "6981006fabd644e39266e0cd90b3df50"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "640b81ee0f95c25c47f3532597056384"
server_key_iv: "c93b2e93f38bc4388ae22b298cbd6c5b"
server_key_version: "encryptionkey@0"
I20260808 22:50:49.836802 3591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:49.862700 3591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:49.864638 3591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:49.864743 3591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:49.864903 3591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:49.865120 3591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:49.865151 3591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.865171 3591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:49.865185 3591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:49.871284 3591 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:39713
I20260808 22:50:49.871356 3804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:39713 every 8 connection(s)
I20260808 22:50:49.871668 3591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:50:49.876562 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 3591
I20260808 22:50:49.876710 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:50:49.876885 22648 external_mini_cluster.cc:1428] Setting key 4e21abc425bfe8766dd9790fbd2f49ae
Aug 08 22:50:49 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229449, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:50:49.880520 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.872366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57047 (local address 127.22.30.62:38483)
0808 22:50:49.872432 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:49.872434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:49.872600 (+ 166us) server_negotiation.cc:407] Connection header received
0808 22:50:49.872984 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:49.872985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:49.872993 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:49.873045 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:49.873853 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.874331 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.875223 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.875462 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.875537 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.877782 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:49.877784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:49.877784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:49.877792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:49.878982 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.879492 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.879493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.879494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.879568 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:49.879861 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:49.879862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:49.879862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:49.879982 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:49.880065 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:49.880355 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 22:50:49.880434 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:50:49.880690 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:49.872317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:57047)
0808 22:50:49.872472 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:49.872507 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:49.872911 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:49.873088 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:49.873090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:49.873600 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:49.873776 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:49.873780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.874413 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.874414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:49.875142 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:49.875144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:49.875557 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:49.875558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:49.875656 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:49.876030 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:49.876036 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:49.877686 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:50:49.879110 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:49.879117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:49.879119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:49.879354 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:49.879664 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:50:49.879665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:50:49.879665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:50:49.879787 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:50:49.880070 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:50:49.880086 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:50:49.880214 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 22:50:49.880319 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 22:50:49.880466 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260808 22:50:49.881173 3807 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38483
I20260808 22:50:49.881278 3807 heartbeater.cc:464] Registering TS with master...
I20260808 22:50:49.881443 3807 heartbeater.cc:511] Master 127.22.30.62:38483 requested a full tablet report, sending...
I20260808 22:50:49.881793 3078 ts_manager.cc:194] Registered new tserver with Master: 6981006fabd644e39266e0cd90b3df50 (127.22.30.3:39713)
I20260808 22:50:49.882169 3078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:57047
I20260808 22:50:49.888685 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab.
Entry for principal oryx/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab.
WARNING: no policy specified for oryx/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab.
Entry for principal oryx/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab.
WARNING: no policy specified for oryx/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab.
Entry for principal oryx/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab.
I20260808 22:50:49.953682 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 3177
I20260808 22:50:49.962831 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 3379
I20260808 22:50:49.976172 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 3591
I20260808 22:50:49.983347 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 3020
I20260808 22:50:49.990298 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38483
--webserver_interface=127.22.30.62
--webserver_port=34101
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:50:50.069962 3822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:50.070173 3822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:50.070211 3822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:50.071331 3822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:50:50.071411 3822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:50.071436 3822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:50.071465 3822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:50:50.071483 3822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:50:50.072842 3822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=34101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:50.073109 3822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:50.073319 3822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:50.077909 3830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.077929 3833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:50.078058 3822 server_base.cc:1034] running on GCE node
W20260808 22:50:50.077913 3831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:50.078353 3822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:50.078640 3822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:50.079805 3822 hybrid_clock.cc:648] HybridClock initialized: now 1786229450079800 us; error 37 us; skew 500 ppm
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229450, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:50.082206 3822 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:50:50.082643 3822 webserver.cc:460] Webserver started at http://127.22.30.62:34101/ using document root <none> and password file <none>
I20260808 22:50:50.082877 3822 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:50.082964 3822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:50.084295 3822 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:50.085036 3852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.085208 3822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:50.085322 3822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "1bccc25095374548bf430d612ef55393"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "e47d0de05fe429ce667abdd45fc8886b"
server_key_iv: "1e65ddd2086193d92a7f52a607e1b7e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:50.085623 3822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:50.096040 3822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:50.097733 3822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:50.097890 3822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:50.101854 3822 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:38483
I20260808 22:50:50.101900 3929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:38483 every 8 connection(s)
I20260808 22:50:50.102260 3822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:50:50.103657 3932 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:50:50.104271 3932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Bootstrap starting.
I20260808 22:50:50.104943 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 3822
I20260808 22:50:50.105230 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:44573
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:38483
--webserver_port=35459
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:50:50.112958 3932 log.cc:824] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Log is configured to *not* fsync() on all Append() calls
I20260808 22:50:50.114480 3932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:50:50.114759 3932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Bootstrap complete.
I20260808 22:50:50.116950 3932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:50.117192 3932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bccc25095374548bf430d612ef55393, State: Initialized, Role: FOLLOWER
I20260808 22:50:50.117419 3932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:50.117518 3932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:50:50.117571 3932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:50:50.117619 3932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:50:50.118496 3932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:50.118619 3932 leader_election.cc:302] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bccc25095374548bf430d612ef55393; no voters:
I20260808 22:50:50.118831 3932 leader_election.cc:288] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:50:50.118865 3940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:50:50.118980 3940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [term 2 LEADER]: Becoming Leader. State: Replica: 1bccc25095374548bf430d612ef55393, State: Running, Role: LEADER
I20260808 22:50:50.119077 3932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:50:50.119060 3940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } }
I20260808 22:50:50.119305 3942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bccc25095374548bf430d612ef55393. Latest consensus state: current_term: 2 leader_uuid: "1bccc25095374548bf430d612ef55393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } } }
I20260808 22:50:50.119447 3942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:50.119298 3941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1bccc25095374548bf430d612ef55393" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bccc25095374548bf430d612ef55393" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38483 } } }
I20260808 22:50:50.119578 3941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393 [sys.catalog]: This master's current role is: LEADER
I20260808 22:50:50.119967 3944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:50:50.120422 3944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:50:50.120565 3944 catalog_manager.cc:1259] Loaded cluster ID: 480aecf0eb044a7092ed046949fcedb3
I20260808 22:50:50.120756 3944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:50:50.121457 3944 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:50:50.121634 3944 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1bccc25095374548bf430d612ef55393: Loaded TSK: 0
I20260808 22:50:50.121831 3944 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:50:50.197494 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:50.197669 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:50.197691 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:50.198829 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:50.198868 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:50.198912 3937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:50:50.200332 3937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.1.keytab
--principal=oryx/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:44573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=35459
--tserver_master_addrs=127.22.30.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:50.200587 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:50.200840 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:50.206423 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.206408 3974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.206409 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:50.206735 3937 server_base.cc:1034] running on GCE node
I20260808 22:50:50.206935 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:50.207213 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:50.208366 3937 hybrid_clock.cc:648] HybridClock initialized: now 1786229450208346 us; error 57 us; skew 500 ppm
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229450, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:50.210979 3937 init.cc:377] Logged in from keytab as oryx/127.22.30.1@KRBTEST.COM (short username oryx)
I20260808 22:50:50.211419 3937 webserver.cc:460] Webserver started at http://127.22.30.1:35459/ using document root <none> and password file <none>
I20260808 22:50:50.211664 3937 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:50.211757 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:50.213107 3937 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:50.213799 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.213984 3937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:50.214074 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "f2ac83c3110d4e55a25b789b54b32665"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0bf0bb51c2c611665f46ce6ebc3db44c"
server_key_iv: "f187db98d145c9901329b718ad6c3376"
server_key_version: "encryptionkey@0"
I20260808 22:50:50.214391 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:50.224306 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:50.226217 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:50.226366 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:50.226572 3937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:50.226866 3937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:50.226920 3937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.226958 3937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:50.226984 3937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.233359 3937 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:44573
I20260808 22:50:50.233407 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:44573 every 8 connection(s)
I20260808 22:50:50.233711 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.241485 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 3937
I20260808 22:50:50.241824 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:45301
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:38483
--webserver_port=40415
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:50:50.244645 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.234391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:47661)
0808 22:50:50.234567 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.234605 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.234969 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.235431 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.235434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.235928 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.236087 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.236091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.237533 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.237533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.238192 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.238194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.238964 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.238965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.239052 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.239553 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.239561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.244456 (+ 4895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
W20260808 22:50:50.244877 4157 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:38483 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.30.62:38483: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
W20260808 22:50:50.245064 4162 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.234479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:47661 (local address 127.22.30.62:38483)
0808 22:50:50.234763 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.234771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.234803 (+ 32us) server_negotiation.cc:407] Connection header received
0808 22:50:50.235166 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.235172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.235222 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.235351 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.236218 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.237438 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.238285 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.238874 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.238982 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.244626 (+ 5644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:47661: BlockingRecv error: recv got EOF from 127.22.30.1:47661 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":69,"threads_started":1}
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.250844 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.245069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:34993)
0808 22:50:50.245415 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.245427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.245494 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.245694 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.245695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.245854 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.245904 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.245906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.246813 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.246814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.247762 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.247765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.248205 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.248206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.248274 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.248695 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.248700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.250741 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":66,"threads_started":1}
W20260808 22:50:50.251125 4162 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.245229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:34993 (local address 127.22.30.62:38483)
0808 22:50:50.245283 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.245285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.245507 (+ 222us) server_negotiation.cc:407] Connection header received
0808 22:50:50.245534 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.245535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.245546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.245654 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.245994 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.246710 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.247864 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.248122 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.248187 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.250840 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:34993: BlockingRecv error: recv got EOF from 127.22.30.1:34993 (error 108)
Metrics: {"server-negotiator.queue_time_us":38,"thread_start_us":69,"threads_started":1}
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.256309 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.251064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:60577)
0808 22:50:50.251232 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.251242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.251331 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.251560 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.251561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.251731 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.251765 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.251767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.252672 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.252673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.253523 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.253525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.254027 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.254028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.254105 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.254506 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.254511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.256258 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20260808 22:50:50.256395 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.251148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60577 (local address 127.22.30.62:38483)
0808 22:50:50.251199 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.251201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.251307 (+ 106us) server_negotiation.cc:407] Connection header received
0808 22:50:50.251434 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.251435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.251445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.251524 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.251855 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.252570 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.253628 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.253934 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.254004 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.256348 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:60577: BlockingRecv error: recv got EOF from 127.22.30.1:60577 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260808 22:50:50.256673 4157 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:50:50.326424 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:50.326608 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:50.326630 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:50.327917 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:50.327957 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:50.327982 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:50:50.329429 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.2.keytab
--principal=oryx/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:45301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=40415
--tserver_master_addrs=127.22.30.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:50.329659 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:50.329875 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:50.334817 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.334817 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.334867 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:50.335095 4163 server_base.cc:1034] running on GCE node
I20260808 22:50:50.335337 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:50.335623 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:50.336809 4163 hybrid_clock.cc:648] HybridClock initialized: now 1786229450336805 us; error 36 us; skew 500 ppm
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229450, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:50.343355 4163 init.cc:377] Logged in from keytab as oryx/127.22.30.2@KRBTEST.COM (short username oryx)
I20260808 22:50:50.343823 4163 webserver.cc:460] Webserver started at http://127.22.30.2:40415/ using document root <none> and password file <none>
I20260808 22:50:50.344053 4163 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:50.344142 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:50.345476 4163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:50.346061 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.346230 4163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:50.346330 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "9d180a496b9e441d8a7a74f28db4d810"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "0a826ceb124e98dfb800e3f33af1a6c6"
server_key_iv: "e008941bd5aa609f3ba7d42f5b57c1e3"
server_key_version: "encryptionkey@0"
I20260808 22:50:50.346666 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:50.357560 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:50.359290 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:50.359525 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:50.359758 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:50.360086 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:50.360145 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.360180 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:50.360208 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.366400 4163 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:45301
I20260808 22:50:50.366446 4363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:45301 every 8 connection(s)
I20260808 22:50:50.366752 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:50:50.368096 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 4163
I20260808 22:50:50.368422 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:39713
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:38483
--webserver_port=44097
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:50:50.375252 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.367443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:47943 (local address 127.22.30.62:38483)
0808 22:50:50.367508 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.367510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.367718 (+ 208us) server_negotiation.cc:407] Connection header received
0808 22:50:50.368259 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.368262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.368288 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.368358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.369375 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.370138 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.371614 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.371790 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.371842 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.375206 (+ 3364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:47943: BlockingRecv error: recv got EOF from 127.22.30.2:47943 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:50:50.375437 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.367450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:47943)
0808 22:50:50.367599 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.367631 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.368113 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.368421 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.368424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.369082 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.369262 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.369277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.370248 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.370250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.371536 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.371539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.371880 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.371881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.372016 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.372459 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.372468 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.375129 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260808 22:50:50.375706 4364 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:38483 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.30.62:38483: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.380560 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.375974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:42737)
0808 22:50:50.376019 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.376029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.376062 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.376172 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.376173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.376313 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.376364 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.376366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.376904 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.376904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.377687 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.377690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.377997 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.377998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.378067 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.378404 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.378409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.380471 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 22:50:50.380611 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.376022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:42737 (local address 127.22.30.62:38483)
0808 22:50:50.376055 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.376056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.376064 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:50.376114 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.376114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.376122 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.376164 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.376433 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.376825 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.377768 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.377925 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.377970 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.380551 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:42737: BlockingRecv error: recv got EOF from 127.22.30.2:42737 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.384945 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.380854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:55325)
0808 22:50:50.380891 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.380900 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.380950 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.381095 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.381096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.381209 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.381233 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.381234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.381835 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.381836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.382326 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.382327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.382672 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.382673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.382733 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.383133 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.383139 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.384870 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 22:50:50.384995 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.380944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:55325 (local address 127.22.30.62:38483)
0808 22:50:50.380979 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.380981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.380988 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:50.381007 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.381007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.381016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.381091 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.381325 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.381736 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.382412 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.382595 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.382673 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.384937 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:55325: BlockingRecv error: recv got EOF from 127.22.30.2:55325 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 22:50:50.385236 4364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:50:50.450476 4367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:50:50.450639 4367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:50:50.450659 4367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:50:50.451694 4367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:50:50.451730 4367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:50:50.451753 4367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:50:50.453526 4367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/krb5kdc/oryx_127.22.30.3.keytab
--principal=oryx/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:39713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=44097
--tserver_master_addrs=127.22.30.62:38483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:50:50.453747 4367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:50:50.453943 4367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:50:50.458516 4375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.458539 4380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:50:50.458537 4377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:50:50.458784 4367 server_base.cc:1034] running on GCE node
I20260808 22:50:50.458981 4367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:50:50.459244 4367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:50:50.460422 4367 hybrid_clock.cc:648] HybridClock initialized: now 1786229450460410 us; error 24 us; skew 500 ppm
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229450, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:50:50.462806 4367 init.cc:377] Logged in from keytab as oryx/127.22.30.3@KRBTEST.COM (short username oryx)
I20260808 22:50:50.463210 4367 webserver.cc:460] Webserver started at http://127.22.30.3:44097/ using document root <none> and password file <none>
I20260808 22:50:50.463524 4367 fs_manager.cc:359] Metadata directory not provided
I20260808 22:50:50.463598 4367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:50:50.464926 4367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:50:50.465531 4390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.465706 4367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:50:50.465791 4367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "6981006fabd644e39266e0cd90b3df50"
format_stamp: "Formatted at 2026-08-08 22:50:49 on dist-test-slave-x4qh"
server_key: "640b81ee0f95c25c47f3532597056384"
server_key_iv: "c93b2e93f38bc4388ae22b298cbd6c5b"
server_key_version: "encryptionkey@0"
I20260808 22:50:50.466058 4367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:50:50.490550 4367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:50:50.492300 4367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:50:50.492403 4367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:50:50.492568 4367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:50:50.492861 4367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:50:50.492894 4367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.492913 4367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:50:50.492962 4367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:50:50.498798 4367 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:39713
I20260808 22:50:50.498842 4564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:39713 every 8 connection(s)
I20260808 22:50:50.499148 4367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.504802 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 4367
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
W20260808 22:50:50.505895 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.499808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57233 (local address 127.22.30.62:38483)
0808 22:50:50.499872 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.499874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.500076 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:50:50.500456 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.500457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.500466 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.500513 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.501325 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.501749 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.502654 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.502835 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.502885 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.505856 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:57233: BlockingRecv error: recv got EOF from 127.22.30.3:57233 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:50:50.505977 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.499803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:57233)
0808 22:50:50.499967 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.499997 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.500376 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.500576 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.500578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.501061 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.501217 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.501221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.501836 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.501837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.502580 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.502582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.502907 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.502908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.503019 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.503469 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.503477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.505788 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
W20260808 22:50:50.506152 4565 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:38483 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.30.62:38483: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.510504 4567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.506356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:57345)
0808 22:50:50.506460 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.506469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.506522 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.506663 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.506664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.506795 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.506833 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.506834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.507463 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.507464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.507929 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.507930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.508337 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.508338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.508400 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.508692 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.508697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.510467 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260808 22:50:50.510637 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.506422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57345 (local address 127.22.30.62:38483)
0808 22:50:50.506457 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.506459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.506511 (+ 52us) server_negotiation.cc:407] Connection header received
0808 22:50:50.506589 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.506590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.506599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.506650 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.506926 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.507357 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.508033 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.508256 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.508325 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.510571 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:57345: BlockingRecv error: recv got EOF from 127.22.30.3:57345 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:50:50.512517 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.505312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.22.30.62:38483)
0808 22:50:50.505353 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.505355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.505453 (+ 98us) server_negotiation.cc:407] Connection header received
0808 22:50:50.505545 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.505546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.505556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.505628 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.506366 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.506837 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.507660 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.507861 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.507937 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.509330 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:50:50.509334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:50:50.509334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:50:50.509344 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:50:50.511256 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:50.511646 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:50.511648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:50.511649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:50.511747 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:50:50.511947 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:50:50.511948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:50:50.511948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:50:50.512133 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:50:50.512225 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:50:50.512371 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 22:50:50.512441 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:50 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:50.515165 4569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:50.510725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:36271)
0808 22:50:50.510868 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:50.510878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:50.510927 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:50.511070 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:50.511070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:50.511196 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:50.511247 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.511248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.511916 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.511917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.512637 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:50.512641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.513129 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.513131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:50.513196 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.513601 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:50.513605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:50.515075 (+ 1470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
W20260808 22:50:50.515195 4166 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:50.510881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36271 (local address 127.22.30.62:38483)
0808 22:50:50.510924 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:50.510925 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:50.510935 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:50:50.510986 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:50.510987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:50.510996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:50.511058 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:50.511313 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.511847 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.512751 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:50.513001 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:50.513102 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:50.515138 (+ 2036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:36271: BlockingRecv error: recv got EOF from 127.22.30.3:36271 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 22:50:50.515326 4565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 22:50:51 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:51 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:51.261106 4571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:51.256974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:54871)
0808 22:50:51.257112 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:51.257126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:51.257171 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:51.257325 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:51.257326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:51.257469 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:51.257535 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:51.257537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.258157 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.258157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:51.258624 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:51.258625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.258956 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.258957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:51.259012 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:51.259528 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:51.259532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:51.261054 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
W20260808 22:50:51.261214 4572 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:51.257055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:54871 (local address 127.22.30.62:38483)
0808 22:50:51.257183 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:51.257184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:51.257215 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:50:51.257235 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:51.257235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:51.257245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:51.257298 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:51.257596 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.258086 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.258700 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.258891 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.258940 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:51.261140 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:54871: BlockingRecv error: recv got EOF from 127.22.30.1:54871 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
Aug 08 22:50:51 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:51 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:51.390005 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:51.385586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:39155)
0808 22:50:51.385726 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:51.385737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:51.385821 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:51.385943 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:51.385943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:51.386097 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:51.386138 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:51.386140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.386756 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.386757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:51.387244 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:51.387246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.387698 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.387699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:51.387762 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:51.388158 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:51.388185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:51.389926 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
W20260808 22:50:51.390080 4572 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:51.385662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:39155 (local address 127.22.30.62:38483)
0808 22:50:51.385729 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:51.385731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:51.385807 (+ 76us) server_negotiation.cc:407] Connection header received
0808 22:50:51.385880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:51.385880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:51.385892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:51.385938 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:51.386212 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.386679 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.387332 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.387626 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.387699 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:51.390022 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:39155: BlockingRecv error: recv got EOF from 127.22.30.2:39155 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 08 22:50:51 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:51 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:51.520133 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:51.515640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:48923)
0808 22:50:51.515786 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:51.515796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:51.515856 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:51.515981 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:51.515982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:51.516137 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:51.516180 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:51.516181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.516857 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.516858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:51.517325 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:51.517327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.517687 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.517688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:51.517763 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:51.518135 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:51.518138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:51.520086 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
W20260808 22:50:51.520270 4572 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:51.515711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:48923 (local address 127.22.30.62:38483)
0808 22:50:51.515779 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:51.515781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:51.515837 (+ 56us) server_negotiation.cc:407] Connection header received
0808 22:50:51.515913 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:51.515913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:51.515925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:51.515967 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:51.516254 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.516779 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.517405 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:51.517614 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:51.517686 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:51.520158 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:48923: BlockingRecv error: recv got EOF from 127.22.30.3:48923 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 22:50:52 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:52 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:52.265781 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:52.261627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:59649)
0808 22:50:52.261760 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:52.261777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:52.261824 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:52.261934 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:52.261935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:52.262071 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:52.262118 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:52.262120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.262698 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.262698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:52.263168 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:52.263170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.263554 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.263554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:52.263598 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:52.264026 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:52.264031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:52.265722 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
W20260808 22:50:52.265885 4577 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:52.261710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:59649 (local address 127.22.30.62:38483)
0808 22:50:52.261828 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:52.261830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:52.261838 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:52.261858 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:52.261859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:52.261868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:52.261919 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:52.262180 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.262630 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.263235 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.263487 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.263538 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:52.265813 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:59649: BlockingRecv error: recv got EOF from 127.22.30.1:59649 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
Aug 08 22:50:52 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:52 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:52.394611 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:52.390504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:53213)
0808 22:50:52.390639 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:52.390649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:52.390701 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:52.390831 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:52.390831 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:52.390986 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:52.391035 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:52.391037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.391652 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.391653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:52.392150 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:52.392151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.392447 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.392448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:52.392486 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:52.392962 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:52.392967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:52.394551 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260808 22:50:52.394708 4577 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:52.390573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:53213 (local address 127.22.30.62:38483)
0808 22:50:52.390637 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:52.390639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:52.390700 (+ 61us) server_negotiation.cc:407] Connection header received
0808 22:50:52.390744 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:52.390745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:52.390755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:52.390810 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:52.391098 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.391590 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.392218 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.392386 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.392432 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:52.394643 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:53213: BlockingRecv error: recv got EOF from 127.22.30.2:53213 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 22:50:52 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:52 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:52.524905 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:52.520602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:58439)
0808 22:50:52.520749 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:52.520761 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:52.520837 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:52.521015 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:52.521016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:52.521173 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:52.521224 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:52.521225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.521826 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.521827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:52.522334 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:52.522336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.522645 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.522646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:52.522685 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:52.523121 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:52.523125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:52.524851 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
W20260808 22:50:52.525054 4577 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:52.520717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:58439 (local address 127.22.30.62:38483)
0808 22:50:52.520788 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:52.520790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:52.520825 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:50:52.520927 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:52.520927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:52.520945 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:52.520992 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:52.521290 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.521755 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.522410 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:52.522582 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:52.522629 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:52.524963 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:58439: BlockingRecv error: recv got EOF from 127.22.30.3:58439 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 22:50:53 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:53 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:53.270823 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:53.266192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:60219)
0808 22:50:53.266371 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:53.266383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:53.266441 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:53.266631 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:53.266631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:53.266826 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:53.266873 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:53.266874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.267541 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.267542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:53.268017 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:53.268018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.268334 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.268335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:53.268428 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:53.268943 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:53.268948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:53.270729 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
W20260808 22:50:53.270889 4584 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:53.266403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60219 (local address 127.22.30.62:38483)
0808 22:50:53.266512 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:53.266514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:53.266522 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:53.266550 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:53.266550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:53.266558 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:53.266612 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:53.266932 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.267480 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.268082 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.268252 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.268324 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:53.270807 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:60219: BlockingRecv error: recv got EOF from 127.22.30.1:60219 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
Aug 08 22:50:53 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:53 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:53.400274 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:53.395271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:39503)
0808 22:50:53.395480 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:53.395492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:53.395559 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:53.395708 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:53.395709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:53.395893 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:53.395940 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:53.395941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.396604 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.396605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:53.397153 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:53.397154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.397529 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.397530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:53.397590 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:53.398134 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:53.398139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:53.400180 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":132,"threads_started":1}
W20260808 22:50:53.400329 4584 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:53.395438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:39503 (local address 127.22.30.62:38483)
0808 22:50:53.395484 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:53.395486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:53.395538 (+ 52us) server_negotiation.cc:407] Connection header received
0808 22:50:53.395616 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:53.395617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:53.395631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:53.395698 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:53.396016 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.396516 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.397241 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.397442 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.397528 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:53.400257 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:39503: BlockingRecv error: recv got EOF from 127.22.30.2:39503 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 08 22:50:53 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:53 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:53.529663 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:53.525327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:58585)
0808 22:50:53.525459 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:53.525469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:53.525526 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:53.525645 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:53.525646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:53.525792 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:53.525840 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:53.525842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.526427 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.526427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:53.526954 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:53.526955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.527438 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.527439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:53.527501 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:53.527923 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:53.527928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:53.529610 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
W20260808 22:50:53.529776 4584 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:53.525376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:58585 (local address 127.22.30.62:38483)
0808 22:50:53.525419 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:53.525421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:53.525525 (+ 104us) server_negotiation.cc:407] Connection header received
0808 22:50:53.525577 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:53.525578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:53.525589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:53.525627 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:53.525898 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.526364 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.527017 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:53.527306 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:53.527400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:53.529693 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:58585: BlockingRecv error: recv got EOF from 127.22.30.3:58585 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 22:50:54 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:54 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:54.275473 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:54.271256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:37535)
0808 22:50:54.271407 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:54.271424 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:54.271471 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:54.271597 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:54.271597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:54.271747 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:54.271809 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:54.271811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.272419 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.272438 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:54.272960 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:54.272961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.273270 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.273271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:54.273312 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:54.273729 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:54.273733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:54.275363 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
W20260808 22:50:54.275514 4589 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:54.271322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:37535 (local address 127.22.30.62:38483)
0808 22:50:54.271471 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:54.271473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:54.271481 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:50:54.271518 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:54.271518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:54.271528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:54.271577 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:54.271871 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.272344 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.273030 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.273205 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.273252 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:54.275452 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:37535: BlockingRecv error: recv got EOF from 127.22.30.1:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
Aug 08 22:50:54 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:54 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:54.405267 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:54.400882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:58629)
0808 22:50:54.401124 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:54.401135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:54.401194 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:54.401349 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:54.401349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:54.401503 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:54.401550 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:54.401551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.402190 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.402190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:54.402698 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:54.402699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.403001 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.403002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:54.403042 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:54.403505 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:54.403509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:54.405210 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
W20260808 22:50:54.405390 4589 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:54.400952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:58629 (local address 127.22.30.62:38483)
0808 22:50:54.401016 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:54.401018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:54.401197 (+ 179us) server_negotiation.cc:407] Connection header received
0808 22:50:54.401242 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:54.401242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:54.401251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:54.401316 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:54.401609 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.402126 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.402766 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.402941 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.402989 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:54.405310 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:58629: BlockingRecv error: recv got EOF from 127.22.30.2:58629 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 08 22:50:54 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:54 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:54.534646 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:54.530235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:56589)
0808 22:50:54.530400 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:54.530410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:54.530466 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:54.530593 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:54.530594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:54.530747 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:54.530801 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:54.530802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.531449 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.531450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:54.531952 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:54.531954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.532248 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.532249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:54.532294 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:54.532806 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:54.532810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:54.534537 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260808 22:50:54.534684 4589 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:54.530322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:56589 (local address 127.22.30.62:38483)
0808 22:50:54.530382 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:54.530384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:54.530467 (+ 83us) server_negotiation.cc:407] Connection header received
0808 22:50:54.530508 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:54.530509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:54.530520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:54.530572 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:54.530861 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.531332 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.532017 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:54.532188 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:54.532236 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:54.534608 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:56589: BlockingRecv error: recv got EOF from 127.22.30.3:56589 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 08 22:50:55 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:55 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:55.280650 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:55.275942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:57353)
0808 22:50:55.276067 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:55.276076 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:55.276127 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:55.276262 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:55.276263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:55.276410 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:55.276461 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:55.276463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.277370 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.277371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:55.277932 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:55.277934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.278250 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.278250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:55.278289 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:55.278677 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:55.278681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:55.280549 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
W20260808 22:50:55.280699 4594 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:55.276028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:57353 (local address 127.22.30.62:38483)
0808 22:50:55.276150 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:55.276152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:55.276159 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:50:55.276179 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:55.276180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:55.276189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:55.276248 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:55.276525 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.277263 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.278008 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.278185 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.278238 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:55.280632 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:57353: BlockingRecv error: recv got EOF from 127.22.30.1:57353 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
Aug 08 22:50:55 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:55 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:55.415956 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:55.405796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:37667)
0808 22:50:55.405952 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:55.405964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:55.406028 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:55.406179 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:55.406180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:55.406378 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:55.406424 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:55.406426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.407115 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.407116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:55.407699 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:55.407700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.408066 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.408067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:55.408129 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:55.414225 (+ 6096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:55.414230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:55.415892 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
W20260808 22:50:55.416046 4594 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:55.405885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:37667 (local address 127.22.30.62:38483)
0808 22:50:55.405949 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:55.405951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:55.406029 (+ 78us) server_negotiation.cc:407] Connection header received
0808 22:50:55.406105 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:55.406106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:55.406118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:55.406164 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:55.406507 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.407015 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.407791 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.407994 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.408069 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:55.415983 (+ 7914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:37667: BlockingRecv error: recv got EOF from 127.22.30.2:37667 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 08 22:50:55 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:55 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:55.539249 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:55.535073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:45293)
0808 22:50:55.535194 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:55.535209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:55.535268 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:55.535463 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:55.535464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:55.535621 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:55.535667 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:55.535669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.536313 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.536314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:55.536879 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:55.536880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.537197 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.537198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:55.537254 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:55.537608 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:55.537612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:55.539185 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
W20260808 22:50:55.539314 4594 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:55.535231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:45293 (local address 127.22.30.62:38483)
0808 22:50:55.535301 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:55.535303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:55.535312 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:50:55.535336 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:55.535337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:55.535352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:55.535450 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:55.535752 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.536205 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.536962 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:55.537123 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:55.537195 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:55.539263 (+ 2068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:45293: BlockingRecv error: recv got EOF from 127.22.30.3:45293 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 22:50:56 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:56 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:56.285633 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:56.281329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:45695)
0808 22:50:56.281478 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:56.281488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:56.281552 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:56.281662 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:56.281663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:56.281820 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:56.281871 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:56.281872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.282513 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.282513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:56.282968 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:56.282969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.283265 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.283265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:56.283327 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:56.283822 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:56.283827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:56.285548 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
W20260808 22:50:56.285686 4601 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:56.281386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:45695 (local address 127.22.30.62:38483)
0808 22:50:56.281498 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:56.281500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:56.281536 (+ 36us) server_negotiation.cc:407] Connection header received
0808 22:50:56.281589 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:56.281590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:56.281599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:56.281647 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:56.281930 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.282456 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.283025 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.283207 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.283251 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:56.285632 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:45695: BlockingRecv error: recv got EOF from 127.22.30.1:45695 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
Aug 08 22:50:56 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:56 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:56.421044 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:56.416417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:57933)
0808 22:50:56.416563 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:56.416574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:56.416660 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:56.416813 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:56.416814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:56.416977 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:56.417022 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:56.417024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.417664 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.417665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:56.418150 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:56.418152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.418518 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.418519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:56.418576 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:56.419019 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:56.419024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:56.420978 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260808 22:50:56.421093 4601 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:56.416504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57933 (local address 127.22.30.62:38483)
0808 22:50:56.416566 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:56.416568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:56.416642 (+ 74us) server_negotiation.cc:407] Connection header received
0808 22:50:56.416722 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:56.416723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:56.416733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:56.416805 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:56.417102 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.417584 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.418239 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.418424 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.418519 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:56.421050 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:57933: BlockingRecv error: recv got EOF from 127.22.30.2:57933 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 08 22:50:56 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:56 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:56.543988 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:56.539783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:47323)
0808 22:50:56.539923 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:56.539933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:56.539988 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:56.540117 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:56.540117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:56.540280 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:56.540343 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:56.540345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.540949 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.540949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:56.541457 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:56.541458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.541758 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.541758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:56.541797 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:56.542242 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:56.542246 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:56.543925 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260808 22:50:56.544083 4601 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:56.539854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:47323 (local address 127.22.30.62:38483)
0808 22:50:56.539898 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:56.539900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:56.539973 (+ 73us) server_negotiation.cc:407] Connection header received
0808 22:50:56.540031 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:56.540031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:56.540042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:56.540095 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:56.540404 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.540886 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.541519 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:56.541699 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:56.541745 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:56.544023 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:47323: BlockingRecv error: recv got EOF from 127.22.30.3:47323 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 22:50:57 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:57 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:57.290354 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:57.286052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:55483)
0808 22:50:57.286186 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:57.286196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:57.286244 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:57.286383 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:57.286384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:57.286534 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:57.286585 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:57.286587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.287174 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.287175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:57.287760 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:57.287762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.288078 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.288079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:57.288122 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:57.288587 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:57.288592 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:57.290298 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
W20260808 22:50:57.290486 4605 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:57.286115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:55483 (local address 127.22.30.62:38483)
0808 22:50:57.286216 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:57.286217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:57.286228 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:50:57.286302 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:57.286303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:57.286313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:57.286371 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:57.286639 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.287114 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.287837 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.288024 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.288071 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:57.290399 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:55483: BlockingRecv error: recv got EOF from 127.22.30.1:55483 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
Aug 08 22:50:57 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:57 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:57.425770 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:57.421445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:56831)
0808 22:50:57.421584 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:57.421595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:57.421667 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:57.421816 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:57.421817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:57.421973 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:57.422018 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:57.422019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.422786 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.422787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:57.423269 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:57.423271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.423623 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.423623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:57.423665 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:57.424106 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:57.424111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:57.425717 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
W20260808 22:50:57.425885 4605 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:57.421507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:56831 (local address 127.22.30.62:38483)
0808 22:50:57.421576 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:57.421578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:57.421652 (+ 74us) server_negotiation.cc:407] Connection header received
0808 22:50:57.421734 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:57.421735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:57.421745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:57.421811 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:57.422095 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.422690 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.423332 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.423561 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.423610 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:57.425809 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:56831: BlockingRecv error: recv got EOF from 127.22.30.2:56831 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 22:50:57 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:57 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:57.548628 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:57.544450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:53009)
0808 22:50:57.544585 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:57.544599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:57.544653 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:57.544775 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:57.544776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:57.544938 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:57.544989 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:57.544990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.545593 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.545593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:57.546103 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:57.546105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.546402 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.546402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:57.546441 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:57.546860 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:57.546865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:57.548571 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
W20260808 22:50:57.548823 4605 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:57.544524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:53009 (local address 127.22.30.62:38483)
0808 22:50:57.544583 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:57.544584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:57.544648 (+ 64us) server_negotiation.cc:407] Connection header received
0808 22:50:57.544694 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:57.544695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:57.544705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:57.544760 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:57.545045 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.545529 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.546167 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:57.546343 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:57.546391 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:57.548764 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:53009: BlockingRecv error: recv got EOF from 127.22.30.3:53009 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 08 22:50:58 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:58 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:58.294842 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:58.290749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:45297)
0808 22:50:58.290900 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:58.290909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:58.290953 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:58.291058 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:58.291059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:58.291201 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:58.291250 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:58.291251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.291829 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.291830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:58.292339 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:58.292340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.292679 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.292679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:58.292718 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:58.293192 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:58.293196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:58.294789 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
W20260808 22:50:58.294951 4611 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:58.290836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:45297 (local address 127.22.30.62:38483)
0808 22:50:58.290941 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:58.290943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:58.290955 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:50:58.290993 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:58.290993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:58.291002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:58.291054 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:58.291300 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.291772 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.292401 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.292620 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.292669 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:58.294872 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:45297: BlockingRecv error: recv got EOF from 127.22.30.1:45297 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
Aug 08 22:50:58 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:58 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:58.430490 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:58.426212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:38987)
0808 22:50:58.426393 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:58.426404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:58.426470 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:58.426579 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:58.426579 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:58.426712 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:58.426773 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:58.426774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.427503 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.427503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:58.427969 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:58.427970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.428417 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.428418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:58.428478 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:58.428837 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:58.428841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:58.430436 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
W20260808 22:50:58.430619 4611 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:58.426371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:38987 (local address 127.22.30.62:38483)
0808 22:50:58.426425 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:58.426427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:58.426455 (+ 28us) server_negotiation.cc:407] Connection header received
0808 22:50:58.426511 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:58.426512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:58.426520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:58.426561 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:58.426830 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.427448 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.428061 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.428336 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.428414 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:58.430533 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:38987: BlockingRecv error: recv got EOF from 127.22.30.2:38987 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 08 22:50:58 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:58 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:58.553165 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:58.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:33499)
0808 22:50:58.549180 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:58.549190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:58.549246 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:58.549362 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:58.549363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:58.549526 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:58.549574 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:58.549575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.550212 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.550212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:58.550718 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:58.550719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.550990 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.550990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:58.551028 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:58.551480 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:58.551484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:58.553113 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
W20260808 22:50:58.553256 4611 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:58.549089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:33499 (local address 127.22.30.62:38483)
0808 22:50:58.549133 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:58.549135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:58.549232 (+ 97us) server_negotiation.cc:407] Connection header received
0808 22:50:58.549291 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:58.549292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:58.549302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:58.549353 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:58.549630 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.550147 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.550778 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:58.550936 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:58.550980 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:58.553194 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:33499: BlockingRecv error: recv got EOF from 127.22.30.3:33499 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 08 22:50:59 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:59 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:59.299305 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:59.295325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:38193)
0808 22:50:59.295463 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:59.295472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:59.295546 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:59.295709 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:59.295710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:59.295910 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:59.295958 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:59.295961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.296508 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.296509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:59.296966 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:59.296967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.297246 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.297247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:59.297291 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:59.297673 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:59.297677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:59.299244 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260808 22:50:59.299431 4616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:59.295395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:38193 (local address 127.22.30.62:38483)
0808 22:50:59.295484 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:59.295486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:59.295537 (+ 51us) server_negotiation.cc:407] Connection header received
0808 22:50:59.295606 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:59.295607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:59.295619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:59.295691 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:59.296017 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.296454 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.297023 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.297193 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.297237 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:59.299334 (+ 2097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:38193: BlockingRecv error: recv got EOF from 127.22.30.1:38193 (error 108)
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":44,"threads_started":1}
Aug 08 22:50:59 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:59 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:59.435509 4617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:59.431043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:49827)
0808 22:50:59.431181 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:59.431191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:59.431243 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:59.431397 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:59.431398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:59.431555 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:59.431603 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:59.431605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.432223 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.432224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:59.432736 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:59.432737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.433030 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.433031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:59.433070 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:59.433568 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:59.433574 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:59.435442 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260808 22:50:59.435573 4616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:59.431122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:49827 (local address 127.22.30.62:38483)
0808 22:50:59.431182 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:59.431183 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:59.431244 (+ 61us) server_negotiation.cc:407] Connection header received
0808 22:50:59.431287 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:59.431288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:59.431298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:59.431347 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:59.431677 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.432163 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.432805 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.432973 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.433020 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:59.435508 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:49827: BlockingRecv error: recv got EOF from 127.22.30.2:49827 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 08 22:50:59 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:50:59 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:50:59.557622 4618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:50:59.553583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:45497)
0808 22:50:59.553714 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 22:50:59.553723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:50:59.553793 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:50:59.553910 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:50:59.553911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:50:59.554056 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:50:59.554106 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:59.554107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.554663 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.554663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:59.555139 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:50:59.555141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.555482 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.555483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:50:59.555526 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:59.555894 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:50:59.555898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:50:59.557567 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
W20260808 22:50:59.557737 4616 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:50:59.553619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:45497 (local address 127.22.30.62:38483)
0808 22:50:59.553674 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 22:50:59.553677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:50:59.553784 (+ 107us) server_negotiation.cc:407] Connection header received
0808 22:50:59.553839 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:50:59.553840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:50:59.553851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:50:59.553894 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:50:59.554153 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.554596 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.555197 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:50:59.555424 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:50:59.555468 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:50:59.557648 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:45497: BlockingRecv error: recv got EOF from 127.22.30.3:45497 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 08 22:51:00 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:00 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:00.303925 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:00.299751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:53049)
0808 22:51:00.299878 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:00.299891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:00.299950 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:00.300074 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:00.300075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:00.300228 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:00.300278 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:00.300280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.300932 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.300933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:00.301425 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:00.301427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.301757 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.301757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:00.301803 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:00.302221 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:00.302225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:00.303870 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260808 22:51:00.304039 4620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:00.299811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:53049 (local address 127.22.30.62:38483)
0808 22:51:00.299923 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:00.299925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:00.299936 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:51:00.299994 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:00.299995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:00.300004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:00.300056 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:00.300340 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.300871 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.301487 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.301703 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.301749 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:00.303959 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:53049: BlockingRecv error: recv got EOF from 127.22.30.1:53049 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
Aug 08 22:51:00 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:00 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:00.445273 4621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:00.435882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:36685)
0808 22:51:00.436024 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:00.436035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:00.436109 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:00.436251 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:00.436251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:00.436426 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:00.436470 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:00.436472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.437246 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.437247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:00.437794 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:00.437796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.438125 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.438126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:00.438177 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:00.443611 (+ 5434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:00.443615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:00.445180 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
W20260808 22:51:00.445530 4620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:00.435942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:36685 (local address 127.22.30.62:38483)
0808 22:51:00.436001 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:00.436003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:00.436090 (+ 87us) server_negotiation.cc:407] Connection header received
0808 22:51:00.436169 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:00.436170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:00.436181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:00.436246 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:00.436541 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.437182 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.437887 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.438065 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.438127 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:00.445313 (+ 7186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:36685: BlockingRecv error: recv got EOF from 127.22.30.2:36685 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 08 22:51:00 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:00 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:00.562625 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:00.558192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:56825)
0808 22:51:00.558442 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:00.558454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:00.558518 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:00.558655 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:00.558656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:00.558810 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:00.558887 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:00.558888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.559530 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.559531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:00.560123 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:00.560125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.560433 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.560434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:00.560472 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:00.560963 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:00.560967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:00.562574 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":65,"threads_started":1}
W20260808 22:51:00.562747 4620 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:00.558417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:56825 (local address 127.22.30.62:38483)
0808 22:51:00.558474 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:00.558475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:00.558504 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:51:00.558564 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:00.558565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:00.558593 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:00.558633 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:00.558950 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.559469 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.560193 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:00.560374 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:00.560424 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:00.562671 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:56825: BlockingRecv error: recv got EOF from 127.22.30.3:56825 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 22:51:01 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:01 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:01.308439 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:01.304331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:47823)
0808 22:51:01.304480 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:01.304489 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:01.304542 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:01.304659 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:01.304660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:01.304817 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:01.304864 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:01.304866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.305484 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.305485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:01.306004 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:01.306006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.306309 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.306310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:01.306347 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:01.306734 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:01.306738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:01.308389 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
W20260808 22:51:01.308561 4627 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:01.304406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:47823 (local address 127.22.30.62:38483)
0808 22:51:01.304496 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:01.304497 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:01.304536 (+ 39us) server_negotiation.cc:407] Connection header received
0808 22:51:01.304584 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:01.304584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:01.304593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:01.304647 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:01.304918 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.305413 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.306065 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.306249 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.306298 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:01.308482 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:47823: BlockingRecv error: recv got EOF from 127.22.30.1:47823 (error 108)
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
Aug 08 22:51:01 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:01 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:01.450003 4630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:01.445902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:36979)
0808 22:51:01.446067 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:01.446076 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:01.446132 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:01.446245 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:01.446246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:01.446399 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:01.446446 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:01.446447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.447009 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.447010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:01.447593 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:01.447596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.447894 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.447895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:01.447944 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:01.448354 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:01.448358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:01.449949 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
W20260808 22:51:01.450126 4627 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:01.445950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:36979 (local address 127.22.30.62:38483)
0808 22:51:01.446001 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:01.446003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:01.446134 (+ 131us) server_negotiation.cc:407] Connection header received
0808 22:51:01.446178 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:01.446178 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:01.446188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:01.446232 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:01.446500 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.446952 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.447657 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.447836 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.447888 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:01.450032 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:36979: BlockingRecv error: recv got EOF from 127.22.30.2:36979 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 08 22:51:01 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:01 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:01.567497 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:01.563128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:39393)
0808 22:51:01.563253 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:01.563263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:01.563324 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:01.563485 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:01.563486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:01.563653 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:01.563696 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:01.563697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.564387 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.564388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:01.564876 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:01.564877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.565243 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.565244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:01.565311 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:01.565734 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:01.565738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:01.567373 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260808 22:51:01.567528 4627 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:01.563285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:39393 (local address 127.22.30.62:38483)
0808 22:51:01.563351 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:01.563353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:01.563361 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:51:01.563403 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:01.563403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:01.563420 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:01.563475 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:01.563770 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.564300 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.564955 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:01.565154 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:01.565240 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:01.567465 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:39393: BlockingRecv error: recv got EOF from 127.22.30.3:39393 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 08 22:51:02 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:02 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:02.313158 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:02.308852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:43651)
0808 22:51:02.309028 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:02.309039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:02.309102 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:02.309235 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:02.309235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:02.309405 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:02.309452 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:02.309454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.310066 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.310067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:02.310614 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:02.310616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.310941 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.310942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:02.310979 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:02.311429 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:02.311433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:02.313093 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
W20260808 22:51:02.313233 4639 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:02.308911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:43651 (local address 127.22.30.62:38483)
0808 22:51:02.309050 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:02.309052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:02.309086 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:51:02.309152 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:02.309152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:02.309162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:02.309217 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:02.309513 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.310005 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.310683 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.310885 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.310933 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:02.313172 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:43651: BlockingRecv error: recv got EOF from 127.22.30.1:43651 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
Aug 08 22:51:02 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:02 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:02.454651 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:02.450404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:55851)
0808 22:51:02.450569 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:02.450580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:02.450648 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:02.450782 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:02.450782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:02.450960 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:02.451008 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:02.451009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.451645 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.451645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:02.452152 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:02.452154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.452427 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.452427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:02.452464 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:02.452860 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:02.452864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:02.454543 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
W20260808 22:51:02.454679 4639 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:02.450494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:55851 (local address 127.22.30.62:38483)
0808 22:51:02.450552 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:02.450554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:02.450628 (+ 74us) server_negotiation.cc:407] Connection header received
0808 22:51:02.450701 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:02.450702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:02.450712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:02.450763 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:02.451066 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.451587 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.452213 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.452372 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.452417 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:02.454618 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:55851: BlockingRecv error: recv got EOF from 127.22.30.2:55851 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 22:51:02 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:02 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:02.572216 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:02.567946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:45557)
0808 22:51:02.568106 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:02.568116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:02.568198 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:02.568363 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:02.568365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:02.568544 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:02.568589 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:02.568591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.569221 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.569222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:02.569697 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:02.569698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.570045 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.570046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:02.570109 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:02.570508 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:02.570513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:02.572160 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
W20260808 22:51:02.572348 4639 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:02.568020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:45557 (local address 127.22.30.62:38483)
0808 22:51:02.568084 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:02.568087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:02.568178 (+ 91us) server_negotiation.cc:407] Connection header received
0808 22:51:02.568253 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:02.568254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:02.568292 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:02.568354 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:02.568674 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.569140 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.569804 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:02.569977 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:02.570047 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:02.572276 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:45557: BlockingRecv error: recv got EOF from 127.22.30.3:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 08 22:51:03 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:03 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:03.317945 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:03.313637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:60887)
0808 22:51:03.313764 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:03.313779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:03.313826 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:03.313965 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:03.313965 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:03.314117 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:03.314166 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:03.314168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.314796 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.314796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:03.315264 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:03.315265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.315661 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.315661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:03.315702 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:03.316145 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:03.316150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:03.317857 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260808 22:51:03.318035 4644 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:03.313688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60887 (local address 127.22.30.62:38483)
0808 22:51:03.313821 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:03.313823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:03.313831 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:51:03.313876 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:03.313877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:03.313886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:03.313935 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:03.314220 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.314718 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.315340 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.315583 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.315641 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:03.317967 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:60887: BlockingRecv error: recv got EOF from 127.22.30.1:60887 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
Aug 08 22:51:03 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:03 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:03.459596 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:03.455154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:55499)
0808 22:51:03.455331 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:03.455347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:03.455417 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:03.455574 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:03.455574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:03.455735 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:03.455808 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:03.455809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.456411 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.456412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:03.456937 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:03.456938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.457239 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.457239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:03.457279 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:03.457759 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:03.457764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:03.459540 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
W20260808 22:51:03.459718 4644 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:03.455284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:55499 (local address 127.22.30.62:38483)
0808 22:51:03.455390 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:03.455393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:03.455406 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:51:03.455470 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:03.455470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:03.455480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:03.455557 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:03.455868 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.456348 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.456996 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.457182 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.457228 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:03.459624 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:55499: BlockingRecv error: recv got EOF from 127.22.30.2:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 22:51:03 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:03 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:03.576756 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:03.572609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:51713)
0808 22:51:03.572735 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:03.572744 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:03.572815 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:03.572939 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:03.572940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:03.573099 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:03.573145 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:03.573146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.573729 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.573729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:03.574264 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:03.574265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.574563 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.574564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:03.574604 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:03.575034 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:03.575038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:03.576703 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
W20260808 22:51:03.576853 4644 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:03.572680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:51713 (local address 127.22.30.62:38483)
0808 22:51:03.572737 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:03.572739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:03.572814 (+ 75us) server_negotiation.cc:407] Connection header received
0808 22:51:03.572870 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:03.572870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:03.572881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:03.572922 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:03.573201 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.573669 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.574330 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:03.574504 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:03.574552 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:03.576791 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:51713: BlockingRecv error: recv got EOF from 127.22.30.3:51713 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 08 22:51:04 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:04 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:04.322984 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:04.318394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:43821)
0808 22:51:04.318544 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:04.318556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:04.318619 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:04.318747 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:04.318748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:04.318943 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:04.318994 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:04.318997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.319650 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.319651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:04.320224 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:04.320225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.320572 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.320573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:04.320615 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:04.321164 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:04.321169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:04.322917 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
W20260808 22:51:04.323073 4649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:04.318445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:43821 (local address 127.22.30.62:38483)
0808 22:51:04.318566 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:04.318567 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:04.318609 (+ 42us) server_negotiation.cc:407] Connection header received
0808 22:51:04.318665 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:04.318666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:04.318675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:04.318733 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:04.319058 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.319562 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.320321 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.320511 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.320564 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:04.323004 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:43821: BlockingRecv error: recv got EOF from 127.22.30.1:43821 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
Aug 08 22:51:04 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:04 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:04.464334 4650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:04.460153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:33291)
0808 22:51:04.460299 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:04.460308 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:04.460361 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:04.460496 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:04.460497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:04.460657 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:04.460704 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:04.460706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.461298 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.461298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:04.461784 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:04.461786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.462061 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.462062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:04.462099 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:04.462477 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:04.462481 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:04.464213 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260808 22:51:04.464375 4649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:04.460234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:33291 (local address 127.22.30.62:38483)
0808 22:51:04.460306 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:04.460309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:04.460358 (+ 49us) server_negotiation.cc:407] Connection header received
0808 22:51:04.460421 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:04.460422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:04.460433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:04.460475 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:04.460760 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.461236 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.461844 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.462007 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.462051 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:04.464300 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:33291: BlockingRecv error: recv got EOF from 127.22.30.2:33291 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 08 22:51:04 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:04 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.3@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:04.581894 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:04.577387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.3:38281)
0808 22:51:04.577550 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:04.577561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:04.577651 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:04.577806 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:04.577806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:04.577981 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:04.578027 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:04.578029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.578642 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.578643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:04.579136 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:04.579137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.579517 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.579518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:04.579575 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:04.580070 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:04.580074 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:04.581797 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
W20260808 22:51:04.581919 4649 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:04.577473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:38281 (local address 127.22.30.62:38483)
0808 22:51:04.577546 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:04.577548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:04.577632 (+ 84us) server_negotiation.cc:407] Connection header received
0808 22:51:04.577716 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:04.577717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:04.577731 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:04.577795 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:04.578098 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.578563 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.579220 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:04.579442 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:04.579506 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:04.581862 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.3:38281: BlockingRecv error: recv got EOF from 127.22.30.3:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 22:51:05 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:05 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.1@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:05.327669 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.323438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.1:33961)
0808 22:51:05.323565 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:05.323574 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:05.323618 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:05.323754 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:05.323755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:05.323930 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:05.323981 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.323983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.324623 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.324623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.325138 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.325139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.325462 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.325462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.325499 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.325940 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:05.325944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:05.327616 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260808 22:51:05.327780 4653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:05.323497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:33961 (local address 127.22.30.62:38483)
0808 22:51:05.323604 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:05.323605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:05.323613 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:51:05.323668 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:05.323669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:05.323677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:05.323741 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:05.324041 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.324560 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.325214 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.325407 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.325452 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.327700 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.1:33961: BlockingRecv error: recv got EOF from 127.22.30.1:33961 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
Aug 08 22:51:05 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
Aug 08 22:51:05 dist-test-slave-x4qh krb5kdc[3005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.30.2@KRBTEST.COM for oryx/127.22.30.62@KRBTEST.COM, Server not found in Kerberos database
I20260808 22:51:05.472183 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.464831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38483 (local address 127.22.30.2:57049)
0808 22:51:05.464949 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:05.464964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:05.465005 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:05.465117 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:05.465117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:05.465265 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:05.465313 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.465314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.465882 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.465883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.466345 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.466346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.466621 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.466622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.466659 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.470172 (+ 3513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:05.470176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:05.472096 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.30.62:38483: Server oryx/127.22.30.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260808 22:51:05.472230 4653 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:05.464925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57049 (local address 127.22.30.62:38483)
0808 22:51:05.464990 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:05.464991 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:05.465000 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:51:05.465050 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:05.465051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:05.465059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:05.465103 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:05.465368 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.465822 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.466405 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.466566 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.466612 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.472170 (+ 5558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.30.2:57049: BlockingRecv error: recv got EOF from 127.22.30.2:57049 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 22:51:05.513383 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 3937
I20260808 22:51:05.547410 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 4163
I20260808 22:51:05.553927 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 4367
I20260808 22:51:05.565279 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 3822
2026-08-08T22:51:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17606 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T22:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:51:05Z Disabled control of system clock
I20260808 22:51:05.596351 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:44253
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:44253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 22:51:05.672498 4661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:05.672660 4661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:05.672678 4661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:05.673714 4661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:51:05.673751 4661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:05.673764 4661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:05.673776 4661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:51:05.673787 4661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:51:05.674991 4661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:44253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:05.675213 4661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:05.675539 4661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:05.679980 4673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:05.680060 4667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:05.680024 4669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:05.680371 4661 server_base.cc:1034] running on GCE node
I20260808 22:51:05.680588 4661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:05.680842 4661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:05.681978 4661 hybrid_clock.cc:648] HybridClock initialized: now 1786229465681949 us; error 44 us; skew 500 ppm
I20260808 22:51:05.683024 4661 webserver.cc:460] Webserver started at http://127.22.30.62:41283/ using document root <none> and password file <none>
I20260808 22:51:05.683233 4661 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:05.683288 4661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:05.683434 4661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:05.684337 4661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "147c50cdee964367a53f63c67f686ddd"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "2c624d8a5fb4858510ac5fecf47a5dab"
server_key_iv: "32446368a0d070c0dcb1cb198f850550"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.684651 4661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "147c50cdee964367a53f63c67f686ddd"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "2c624d8a5fb4858510ac5fecf47a5dab"
server_key_iv: "32446368a0d070c0dcb1cb198f850550"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.685750 4661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:51:05.686363 4679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.686517 4661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:05.686605 4661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "147c50cdee964367a53f63c67f686ddd"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "2c624d8a5fb4858510ac5fecf47a5dab"
server_key_iv: "32446368a0d070c0dcb1cb198f850550"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.686673 4661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:05.703297 4661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:05.703574 4661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:05.703706 4661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:05.707221 4661 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:44253
I20260808 22:51:05.707252 4758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:44253 every 8 connection(s)
I20260808 22:51:05.707551 4661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:51:05.708169 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:05.710269 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 4661
I20260808 22:51:05.710352 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:51:05.710357 4759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd: Bootstrap starting.
I20260808 22:51:05.710530 22648 external_mini_cluster.cc:1428] Setting key 064867a0759eafaf3a8675c6de507781
I20260808 22:51:05.711061 4759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:05.711436 4759 log.cc:824] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:05.712100 4759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd: No bootstrap required, opened a new log
I20260808 22:51:05.713855 4759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c50cdee964367a53f63c67f686ddd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44253 } }
I20260808 22:51:05.713960 4759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:05.713979 4759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147c50cdee964367a53f63c67f686ddd, State: Initialized, Role: FOLLOWER
I20260808 22:51:05.714089 4759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c50cdee964367a53f63c67f686ddd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44253 } }
I20260808 22:51:05.714360 4759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:51:05.714394 4759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:51:05.714433 4759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:05.714984 4759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c50cdee964367a53f63c67f686ddd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44253 } }
I20260808 22:51:05.715077 4759 leader_election.cc:302] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 147c50cdee964367a53f63c67f686ddd; no voters:
I20260808 22:51:05.715217 4759 leader_election.cc:288] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:51:05.715270 4767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:51:05.715449 4767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [term 1 LEADER]: Becoming Leader. State: Replica: 147c50cdee964367a53f63c67f686ddd, State: Running, Role: LEADER
I20260808 22:51:05.715485 4759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:51:05.715536 4767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c50cdee964367a53f63c67f686ddd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44253 } }
I20260808 22:51:05.715829 4767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 147c50cdee964367a53f63c67f686ddd. Latest consensus state: current_term: 1 leader_uuid: "147c50cdee964367a53f63c67f686ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c50cdee964367a53f63c67f686ddd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44253 } } }
I20260808 22:51:05.715835 4770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "147c50cdee964367a53f63c67f686ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "147c50cdee964367a53f63c67f686ddd" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44253 } } }
I20260808 22:51:05.716149 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.710958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.22.30.62:44253)
0808 22:51:05.711167 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:05.711177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:05.711220 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:51:05.711742 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:05.711749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:05.711819 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:05.711947 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:05.712137 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.713518 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.714313 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.714836 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.714954 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.715469 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:05.715472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:05.715472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:05.715640 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:05.715681 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:05.715695 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:05.715755 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:05.715874 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 22:51:05.715945 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260808 22:51:05.716001 4770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:05.716454 4767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:05.716681 4787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:51:05.717177 4787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:51:05.718529 4787 catalog_manager.cc:1347] Generated new cluster ID: 209e70a2463d429394bf2f4f8dcb0c19
I20260808 22:51:05.718598 4787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:51:05.749987 4787 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:51:05.750541 4787 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:51:05.753898 4787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 147c50cdee964367a53f63c67f686ddd: Generated new TSK 0
I20260808 22:51:05.754047 4787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:51:05.761143 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44253
--builtin_ntp_servers=127.22.30.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 22:51:05.838533 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:05.838716 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:05.838734 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:05.839819 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:05.839883 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:05.839908 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:51:05.841213 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:05.841372 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:05.841652 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:05.846181 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:05.846167 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:05.846220 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:05.846369 4802 server_base.cc:1034] running on GCE node
I20260808 22:51:05.846573 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:05.846794 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:05.847961 4802 hybrid_clock.cc:648] HybridClock initialized: now 1786229465847948 us; error 39 us; skew 500 ppm
I20260808 22:51:05.849143 4802 webserver.cc:460] Webserver started at http://127.22.30.1:38705/ using document root <none> and password file <none>
I20260808 22:51:05.849347 4802 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:05.849390 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:05.849542 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:05.850481 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "01c0d70e1f664f18b68043884c6d605e"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "eaa846c3d44b93fea82bdd7ec2f7e53c"
server_key_iv: "ae0334d5aa9063e4931b9d13d93dddd4"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.850813 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "01c0d70e1f664f18b68043884c6d605e"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "eaa846c3d44b93fea82bdd7ec2f7e53c"
server_key_iv: "ae0334d5aa9063e4931b9d13d93dddd4"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.852053 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:51:05.852717 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.852874 4802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:05.852969 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "01c0d70e1f664f18b68043884c6d605e"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "eaa846c3d44b93fea82bdd7ec2f7e53c"
server_key_iv: "ae0334d5aa9063e4931b9d13d93dddd4"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.853035 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:05.863334 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:05.863603 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:05.863746 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:05.863945 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:05.864235 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:05.864289 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.864346 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:05.864387 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.869700 4802 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:41741
I20260808 22:51:05.869740 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:41741 every 8 connection(s)
I20260808 22:51:05.870002 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:51:05.874220 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.870587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:58765 (local address 127.22.30.62:44253)
0808 22:51:05.870660 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:05.870662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:05.870898 (+ 236us) server_negotiation.cc:407] Connection header received
0808 22:51:05.871302 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:05.871303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:05.871317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:05.871364 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:05.871952 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.872376 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.873219 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.873387 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.873431 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.874037 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:05.874038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:05.874038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:05.874042 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:05.874053 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:05.874056 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:05.874091 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:05.874149 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 22:51:05.874169 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:51:05.874421 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.870552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44253 (local address 127.22.30.1:58765)
0808 22:51:05.870775 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:05.870812 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:05.871203 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:05.871454 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:05.871458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:05.871470 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:05.871890 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.871893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.872443 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.872444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.873152 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.873154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.873449 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.873450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.873544 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.873926 (+ 382us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:51:05.873926 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:05.873949 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:51:05.873968 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:05.874098 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:05.874099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:05.874101 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:51:05.874140 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 22:51:05.874202 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260808 22:51:05.874869 4985 heartbeater.cc:347] Connected to a master server at 127.22.30.62:44253
I20260808 22:51:05.874956 4985 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:05.875098 4985 heartbeater.cc:511] Master 127.22.30.62:44253 requested a full tablet report, sending...
I20260808 22:51:05.875551 4700 ts_manager.cc:194] Registered new tserver with Master: 01c0d70e1f664f18b68043884c6d605e (127.22.30.1:41741)
I20260808 22:51:05.876044 4700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.1:58765
I20260808 22:51:05.876197 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 4802
I20260808 22:51:05.876286 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:51:05.876474 22648 external_mini_cluster.cc:1428] Setting key c0826ce9fe61b9d48201f754e8ddcf16
I20260808 22:51:05.877265 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44253
--builtin_ntp_servers=127.22.30.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 22:51:05.951372 4991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:05.951560 4991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:05.951579 4991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:05.952628 4991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:05.952662 4991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:05.952684 4991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:51:05.953869 4991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:05.954067 4991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:05.954329 4991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:05.958899 5003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:05.958922 5001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:05.959061 4998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:05.959064 4991 server_base.cc:1034] running on GCE node
I20260808 22:51:05.959281 4991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:05.959517 4991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:05.960646 4991 hybrid_clock.cc:648] HybridClock initialized: now 1786229465960631 us; error 29 us; skew 500 ppm
I20260808 22:51:05.961831 4991 webserver.cc:460] Webserver started at http://127.22.30.2:35063/ using document root <none> and password file <none>
I20260808 22:51:05.962044 4991 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:05.962102 4991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:05.962239 4991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:05.963140 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "421c50531e8c4cae8f1d4a68ed54e60b"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "56e7e1497ac99e6e1f80fb32d5fcb245"
server_key_iv: "bba952c4442c244a0c376a919e977d7c"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.963490 4991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "421c50531e8c4cae8f1d4a68ed54e60b"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "56e7e1497ac99e6e1f80fb32d5fcb245"
server_key_iv: "bba952c4442c244a0c376a919e977d7c"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.964674 4991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:51:05.965344 5010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.965505 4991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:05.965596 4991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "421c50531e8c4cae8f1d4a68ed54e60b"
format_stamp: "Formatted at 2026-08-08 22:51:05 on dist-test-slave-x4qh"
server_key: "56e7e1497ac99e6e1f80fb32d5fcb245"
server_key_iv: "bba952c4442c244a0c376a919e977d7c"
server_key_version: "encryptionkey@0"
I20260808 22:51:05.965674 4991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:05.974481 4991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:05.974709 4991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:05.974854 4991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:05.975036 4991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:05.975303 4991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:05.975351 4991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.975404 4991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:05.975430 4991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:05.980954 4991 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:34009
I20260808 22:51:05.980995 5185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:34009 every 8 connection(s)
I20260808 22:51:05.981273 4991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:51:05.981886 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 4991
I20260808 22:51:05.981985 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:51:05.982182 22648 external_mini_cluster.cc:1428] Setting key 7ccdcb6350e3b44435aad118ffd6986f
I20260808 22:51:05.983131 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44253
--builtin_ntp_servers=127.22.30.20:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 22:51:05.986313 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.981904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:55703 (local address 127.22.30.62:44253)
0808 22:51:05.981973 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:05.981976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:05.982151 (+ 175us) server_negotiation.cc:407] Connection header received
0808 22:51:05.982665 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:05.982666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:05.982680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:05.982749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:05.983481 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.984156 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.985065 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.985259 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.985328 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.986030 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:05.986031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:05.986032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:05.986036 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:05.986051 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:05.986055 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:05.986096 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:05.986216 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 22:51:05.986241 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:51:05.986516 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:05.981876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44253 (local address 127.22.30.2:55703)
0808 22:51:05.982044 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:05.982082 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:05.982580 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:05.982837 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:05.982841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:05.982851 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:05.983366 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.983400 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.984237 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.984238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.984980 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:05.984982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:05.985334 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:05.985336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:05.985454 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:05.985911 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:51:05.985912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:05.985934 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:51:05.985952 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:05.986095 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:05.986112 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:05.986114 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:51:05.986211 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 22:51:05.986314 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260808 22:51:05.986925 5186 heartbeater.cc:347] Connected to a master server at 127.22.30.62:44253
I20260808 22:51:05.987025 5186 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:05.987186 5186 heartbeater.cc:511] Master 127.22.30.62:44253 requested a full tablet report, sending...
I20260808 22:51:05.987555 4700 ts_manager.cc:194] Registered new tserver with Master: 421c50531e8c4cae8f1d4a68ed54e60b (127.22.30.2:34009)
I20260808 22:51:05.987892 4700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.2:55703
W20260808 22:51:06.061406 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:06.061589 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:06.061606 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:06.062717 5191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:06.062754 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:06.062776 5191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:51:06.064226 5191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:06.064464 5191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:06.064751 5191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:06.069334 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.069344 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.069415 5202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.069649 5191 server_base.cc:1034] running on GCE node
I20260808 22:51:06.069844 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:06.070078 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:06.071242 5191 hybrid_clock.cc:648] HybridClock initialized: now 1786229466071230 us; error 28 us; skew 500 ppm
I20260808 22:51:06.072440 5191 webserver.cc:460] Webserver started at http://127.22.30.3:42869/ using document root <none> and password file <none>
I20260808 22:51:06.072674 5191 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:06.072737 5191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:06.072871 5191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:06.073810 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "8e687e25d0e24f1082e3780dbf4b2bfd"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "7b7d63f3bde61482379fa8a62f9821d7"
server_key_iv: "e49c5fe9475ca8d6f4b8644227c9dd17"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.074150 5191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "8e687e25d0e24f1082e3780dbf4b2bfd"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "7b7d63f3bde61482379fa8a62f9821d7"
server_key_iv: "e49c5fe9475ca8d6f4b8644227c9dd17"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.075435 5191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:51:06.076151 5211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.076327 5191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:06.076419 5191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "8e687e25d0e24f1082e3780dbf4b2bfd"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "7b7d63f3bde61482379fa8a62f9821d7"
server_key_iv: "e49c5fe9475ca8d6f4b8644227c9dd17"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.076498 5191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:06.089882 5191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:06.090130 5191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:06.090270 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:06.090461 5191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:06.090739 5191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:06.090786 5191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.090826 5191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:06.090857 5191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.096328 5191 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:43169
I20260808 22:51:06.096386 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:43169 every 8 connection(s)
I20260808 22:51:06.096664 5191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:51:06.098315 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 5191
I20260808 22:51:06.098392 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:51:06.098554 22648 external_mini_cluster.cc:1428] Setting key 515749d997cc3ea81db5828c05b20bfd
I20260808 22:51:06.101504 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.097364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:53457 (local address 127.22.30.62:44253)
0808 22:51:06.097422 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.097424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.097648 (+ 224us) server_negotiation.cc:407] Connection header received
0808 22:51:06.098070 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.098071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.098081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.098118 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.098835 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:06.099306 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:06.100263 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:06.100471 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:06.100545 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:06.101241 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:06.101242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:06.101242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:06.101247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:06.101259 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:06.101263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:06.101312 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:06.101402 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 22:51:06.101428 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:51:06.101704 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.097356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:44253 (local address 127.22.30.3:53457)
0808 22:51:06.097545 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:06.097573 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:06.097983 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:06.098200 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:06.098202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:06.098209 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:06.098734 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:06.098739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:06.099416 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:06.099417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:06.100171 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:06.100174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:06.100555 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:06.100557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:06.100692 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:06.101099 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:51:06.101099 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:06.101125 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:51:06.101145 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:06.101317 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:06.101322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:06.101325 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:51:06.101404 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 22:51:06.101499 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260808 22:51:06.102123 5382 heartbeater.cc:347] Connected to a master server at 127.22.30.62:44253
I20260808 22:51:06.102190 5382 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:06.102368 5382 heartbeater.cc:511] Master 127.22.30.62:44253 requested a full tablet report, sending...
I20260808 22:51:06.102680 4700 ts_manager.cc:194] Registered new tserver with Master: 8e687e25d0e24f1082e3780dbf4b2bfd (127.22.30.3:43169)
I20260808 22:51:06.103013 4700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.3:53457
I20260808 22:51:06.110129 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 22:51:06.111954 4762 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:06.111471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.22.30.62:44253)
0808 22:51:06.111522 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.111523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.111680 (+ 157us) server_negotiation.cc:407] Connection header received
0808 22:51:06.111716 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.111717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.111724 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.111773 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.111889 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45058: BlockingRecv error: recv got EOF from 127.0.0.1:45058 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 22:51:06.111994 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 4802
I20260808 22:51:06.131747 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 4991
I20260808 22:51:06.146101 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 5191
I20260808 22:51:06.156669 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 4661
2026-08-08T22:51:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (592 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T22:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:51:06Z Disabled control of system clock
I20260808 22:51:06.188369 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37143
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:51:06.262352 5397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:06.262514 5397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:06.262533 5397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:06.263674 5397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:51:06.263712 5397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:06.263725 5397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:06.263738 5397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:51:06.263780 5397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:51:06.265067 5397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:06.265307 5397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:06.265555 5397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:06.270026 5405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.270071 5408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.270171 5397 server_base.cc:1034] running on GCE node
W20260808 22:51:06.270073 5404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.270498 5397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:06.270779 5397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:06.271936 5397 hybrid_clock.cc:648] HybridClock initialized: now 1786229466271915 us; error 32 us; skew 500 ppm
I20260808 22:51:06.273041 5397 webserver.cc:460] Webserver started at http://127.22.30.62:35527/ using document root <none> and password file <none>
I20260808 22:51:06.273258 5397 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:06.273329 5397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:06.273470 5397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:06.274410 5397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "7580573bd1194fbf9fc256130e66d74a"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "00c2eda454b730f9b25cd61f3d21d677"
server_key_iv: "52f6e864fd4843a600cc14cd99331645"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.274741 5397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "7580573bd1194fbf9fc256130e66d74a"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "00c2eda454b730f9b25cd61f3d21d677"
server_key_iv: "52f6e864fd4843a600cc14cd99331645"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.276108 5397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:51:06.276750 5415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.276911 5397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:06.277002 5397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "7580573bd1194fbf9fc256130e66d74a"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "00c2eda454b730f9b25cd61f3d21d677"
server_key_iv: "52f6e864fd4843a600cc14cd99331645"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.277086 5397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:06.280023 5397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:06.280272 5397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:06.280414 5397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:06.284075 5397 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:37143
I20260808 22:51:06.284106 5494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:37143 every 8 connection(s)
I20260808 22:51:06.284421 5397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:51:06.285048 5495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:06.287315 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a: Bootstrap starting.
I20260808 22:51:06.287828 5495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:06.288035 5495 log.cc:824] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:06.288631 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a: No bootstrap required, opened a new log
I20260808 22:51:06.290100 5495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7580573bd1194fbf9fc256130e66d74a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37143 } }
I20260808 22:51:06.290218 5495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:06.290239 5495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7580573bd1194fbf9fc256130e66d74a, State: Initialized, Role: FOLLOWER
I20260808 22:51:06.290364 5495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7580573bd1194fbf9fc256130e66d74a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37143 } }
I20260808 22:51:06.290442 5495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:51:06.290465 5495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:51:06.290499 5495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:06.291185 5495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7580573bd1194fbf9fc256130e66d74a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37143 } }
I20260808 22:51:06.291296 5495 leader_election.cc:302] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7580573bd1194fbf9fc256130e66d74a; no voters:
I20260808 22:51:06.291451 5495 leader_election.cc:288] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:51:06.291491 5499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:51:06.291636 5499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [term 1 LEADER]: Becoming Leader. State: Replica: 7580573bd1194fbf9fc256130e66d74a, State: Running, Role: LEADER
I20260808 22:51:06.291692 5495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:51:06.291772 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7580573bd1194fbf9fc256130e66d74a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37143 } }
I20260808 22:51:06.292107 5499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7580573bd1194fbf9fc256130e66d74a. Latest consensus state: current_term: 1 leader_uuid: "7580573bd1194fbf9fc256130e66d74a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7580573bd1194fbf9fc256130e66d74a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37143 } } }
I20260808 22:51:06.292428 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7580573bd1194fbf9fc256130e66d74a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7580573bd1194fbf9fc256130e66d74a" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37143 } } }
I20260808 22:51:06.292497 5499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:06.292538 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:06.292963 5514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:51:06.293380 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 5397
I20260808 22:51:06.293481 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:51:06.293609 5514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:51:06.293666 22648 external_mini_cluster.cc:1428] Setting key 2ae8c78e7e9d1ad39876fc35170bfc5d
I20260808 22:51:06.295629 5514 catalog_manager.cc:1347] Generated new cluster ID: c1d061672f0c4f4a8b1352093086cb73
I20260808 22:51:06.295717 5514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:51:06.296020 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.294089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.22.30.62:37143)
0808 22:51:06.294492 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.294500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.294526 (+ 26us) server_negotiation.cc:407] Connection header received
0808 22:51:06.294627 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.294633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.294676 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.294833 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.295511 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:06.295514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:06.295514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:06.295519 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:06.295578 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:06.295591 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:06.295649 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:06.295839 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 22:51:06.295894 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":61,"threads_started":1}
I20260808 22:51:06.315018 5514 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:51:06.315148 5514 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:51:06.323066 5514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7580573bd1194fbf9fc256130e66d74a: Generated new TSK 0
I20260808 22:51:06.323261 5514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:51:06.332958 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37143
--builtin_ntp_servers=127.22.30.20:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:51:06.410012 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:06.410215 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:06.410236 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:06.411353 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:06.411441 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:06.411468 5530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:51:06.412674 5530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:06.412921 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:06.413192 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:06.417691 5538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.417687 5537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.417742 5540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.417785 5530 server_base.cc:1034] running on GCE node
I20260808 22:51:06.418167 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:06.418438 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:06.419602 5530 hybrid_clock.cc:648] HybridClock initialized: now 1786229466419591 us; error 27 us; skew 500 ppm
I20260808 22:51:06.420768 5530 webserver.cc:460] Webserver started at http://127.22.30.1:33759/ using document root <none> and password file <none>
I20260808 22:51:06.421012 5530 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:06.421077 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:06.421206 5530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:06.422139 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "71d685eb51c84c779696b3cf9b1607f6"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "a938be76832142bf1f6f2e49d2c38865"
server_key_iv: "1d05e2a8e711a9f9757936d2ee6de23e"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.422499 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "71d685eb51c84c779696b3cf9b1607f6"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "a938be76832142bf1f6f2e49d2c38865"
server_key_iv: "1d05e2a8e711a9f9757936d2ee6de23e"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.423938 5530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:51:06.424599 5552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.424763 5530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:51:06.424861 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "71d685eb51c84c779696b3cf9b1607f6"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "a938be76832142bf1f6f2e49d2c38865"
server_key_iv: "1d05e2a8e711a9f9757936d2ee6de23e"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.424947 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:06.428069 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:06.428311 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:06.428434 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:06.428619 5530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:06.428893 5530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:06.428946 5530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.428980 5530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:06.429005 5530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.434252 5530 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:40065
I20260808 22:51:06.434312 5721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:40065 every 8 connection(s)
I20260808 22:51:06.434583 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:51:06.437124 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.435208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:48853 (local address 127.22.30.62:37143)
0808 22:51:06.435243 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.435245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.435435 (+ 190us) server_negotiation.cc:407] Connection header received
0808 22:51:06.435828 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.435829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.435836 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.435880 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.436893 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:06.436895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:06.436896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:06.436900 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:06.436915 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:06.436919 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:06.436983 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:06.437049 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 22:51:06.437063 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:51:06.437288 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.435184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37143 (local address 127.22.30.1:48853)
0808 22:51:06.435324 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:06.435349 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:06.435731 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:06.435935 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:06.435938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:06.435946 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:06.436744 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:51:06.436744 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:06.436773 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:51:06.436796 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:06.436980 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:06.436984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:06.436987 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:51:06.437057 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:51:06.437108 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260808 22:51:06.437876 5722 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37143
I20260808 22:51:06.437886 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 5530
I20260808 22:51:06.437974 5722 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:06.437981 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:51:06.438129 5722 heartbeater.cc:511] Master 127.22.30.62:37143 requested a full tablet report, sending...
I20260808 22:51:06.438192 22648 external_mini_cluster.cc:1428] Setting key 8312945ca90b689535450463f8e9a24f
I20260808 22:51:06.438575 5432 ts_manager.cc:194] Registered new tserver with Master: 71d685eb51c84c779696b3cf9b1607f6 (127.22.30.1:40065)
I20260808 22:51:06.439188 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37143
--builtin_ntp_servers=127.22.30.20:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:51:06.512430 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:06.512605 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:06.512638 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:06.513731 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:06.513783 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:06.513821 5726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:51:06.514930 5726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:06.515136 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:06.515362 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:06.519672 5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.519672 5735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.519820 5726 server_base.cc:1034] running on GCE node
W20260808 22:51:06.519663 5737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.520027 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:06.520252 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:06.521381 5726 hybrid_clock.cc:648] HybridClock initialized: now 1786229466521372 us; error 27 us; skew 500 ppm
I20260808 22:51:06.522394 5726 webserver.cc:460] Webserver started at http://127.22.30.2:40567/ using document root <none> and password file <none>
I20260808 22:51:06.522562 5726 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:06.522596 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:06.522667 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:06.523556 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "b8fc57ab1ace47d9a5dd84e3c6b6561a"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "9dd212a320b4dd6165e5503d2da7e6ea"
server_key_iv: "5bc718c03fe2e118dac68fc727d2c982"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.523828 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "b8fc57ab1ace47d9a5dd84e3c6b6561a"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "9dd212a320b4dd6165e5503d2da7e6ea"
server_key_iv: "5bc718c03fe2e118dac68fc727d2c982"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.524974 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:51:06.525523 5745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.525664 5726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:06.525724 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "b8fc57ab1ace47d9a5dd84e3c6b6561a"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "9dd212a320b4dd6165e5503d2da7e6ea"
server_key_iv: "5bc718c03fe2e118dac68fc727d2c982"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.525789 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:06.528812 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:06.528985 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:06.529065 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:06.529202 5726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:06.529403 5726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:06.529429 5726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.529445 5726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:06.529469 5726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.534492 5726 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:35947
I20260808 22:51:06.534549 5911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:35947 every 8 connection(s)
I20260808 22:51:06.534888 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:51:06.537238 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.535503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:36035 (local address 127.22.30.62:37143)
0808 22:51:06.535546 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.535549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.535757 (+ 208us) server_negotiation.cc:407] Connection header received
0808 22:51:06.536128 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.536129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.536134 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.536197 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.537077 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:06.537078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:06.537078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:06.537081 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:06.537093 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:06.537097 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:06.537142 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:06.537184 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 22:51:06.537194 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:51:06.537386 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.535506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37143 (local address 127.22.30.2:36035)
0808 22:51:06.535666 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:06.535692 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:06.536035 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:06.536239 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:06.536242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:06.536254 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:06.536965 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:51:06.536966 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:06.536987 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:51:06.537009 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:06.537143 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:06.537144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:06.537146 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:51:06.537189 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 22:51:06.537226 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260808 22:51:06.537801 5912 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37143
I20260808 22:51:06.537884 5912 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:06.538004 5912 heartbeater.cc:511] Master 127.22.30.62:37143 requested a full tablet report, sending...
I20260808 22:51:06.538302 5432 ts_manager.cc:194] Registered new tserver with Master: b8fc57ab1ace47d9a5dd84e3c6b6561a (127.22.30.2:35947)
I20260808 22:51:06.543749 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 5726
I20260808 22:51:06.543865 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:51:06.544032 22648 external_mini_cluster.cc:1428] Setting key b7f838890a9ef74b4fcf7a17078dccc0
I20260808 22:51:06.544829 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37143
--builtin_ntp_servers=127.22.30.20:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 22:51:06.618131 5918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:06.618276 5918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:06.618295 5918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:06.619333 5918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:06.619381 5918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:06.619405 5918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:51:06.620638 5918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:37143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:06.620864 5918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:06.621156 5918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:06.625551 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.625567 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:06.625547 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:06.625802 5918 server_base.cc:1034] running on GCE node
I20260808 22:51:06.626003 5918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:06.626237 5918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:06.627434 5918 hybrid_clock.cc:648] HybridClock initialized: now 1786229466627430 us; error 43 us; skew 500 ppm
I20260808 22:51:06.628787 5918 webserver.cc:460] Webserver started at http://127.22.30.3:34313/ using document root <none> and password file <none>
I20260808 22:51:06.629050 5918 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:06.629108 5918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:06.629231 5918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:06.630129 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "74fa509a676e4bf5ab21cad1ea7f8dcc"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "f1119c5a655c94aacb453bb9f489f2e2"
server_key_iv: "325c7314af6587467ceefaea74ebe2e4"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.630463 5918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "74fa509a676e4bf5ab21cad1ea7f8dcc"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "f1119c5a655c94aacb453bb9f489f2e2"
server_key_iv: "325c7314af6587467ceefaea74ebe2e4"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.631714 5918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:51:06.632351 5941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.632499 5918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:06.632582 5918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "74fa509a676e4bf5ab21cad1ea7f8dcc"
format_stamp: "Formatted at 2026-08-08 22:51:06 on dist-test-slave-x4qh"
server_key: "f1119c5a655c94aacb453bb9f489f2e2"
server_key_iv: "325c7314af6587467ceefaea74ebe2e4"
server_key_version: "encryptionkey@0"
I20260808 22:51:06.632655 5918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:06.635847 5918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:06.636071 5918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:06.636189 5918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:06.636376 5918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:06.636632 5918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:06.636687 5918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.636727 5918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:06.636752 5918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:06.642140 5918 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:33263
I20260808 22:51:06.642182 6111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:33263 every 8 connection(s)
I20260808 22:51:06.642458 5918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:51:06.644829 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.643072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36001 (local address 127.22.30.62:37143)
0808 22:51:06.643119 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.643122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.643314 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:51:06.643727 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.643729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.643734 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.643792 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.644670 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:06.644671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:06.644671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:51:06.644675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:06.644687 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:51:06.644690 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:06.644732 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:06.644772 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 22:51:06.644783 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:51:06.644990 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:06.643056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37143 (local address 127.22.30.3:36001)
0808 22:51:06.643217 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:06.643242 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:06.643649 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:06.643835 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:06.643837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:06.643844 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:06.644552 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:51:06.644553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:06.644575 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:51:06.644598 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:06.644730 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:06.644732 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:06.644734 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:51:06.644782 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 22:51:06.644821 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260808 22:51:06.645417 6112 heartbeater.cc:347] Connected to a master server at 127.22.30.62:37143
I20260808 22:51:06.645494 6112 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:06.645587 6112 heartbeater.cc:511] Master 127.22.30.62:37143 requested a full tablet report, sending...
I20260808 22:51:06.645880 5432 ts_manager.cc:194] Registered new tserver with Master: 74fa509a676e4bf5ab21cad1ea7f8dcc (127.22.30.3:33263)
I20260808 22:51:06.649658 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 5918
I20260808 22:51:06.649734 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:51:06.649881 22648 external_mini_cluster.cc:1428] Setting key db3bb6704f76be80e16f1193dea3d8c8
I20260808 22:51:06.650665 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 22:51:06.652240 5528 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:51:06.651845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.22.30.62:37143)
0808 22:51:06.651882 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:06.651884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:06.652009 (+ 125us) server_negotiation.cc:407] Connection header received
0808 22:51:06.652044 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:06.652044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:06.652048 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:06.652087 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:06.652191 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51884: BlockingRecv error: recv got EOF from 127.0.0.1:51884 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:51:06.652321 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 5530
I20260808 22:51:06.680641 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 5726
I20260808 22:51:06.693048 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 5918
I20260808 22:51:06.699935 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 5397
2026-08-08T22:51:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (542 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:51:06 dist-test-slave-x4qh krb5kdc[6126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:51:06 dist-test-slave-x4qh krb5kdc[6126](info): set up 2 sockets
Aug 08 22:51:06 dist-test-slave-x4qh krb5kdc[6126](info): commencing operation
krb5kdc: starting...
W20260808 22:51:07.869951 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:51:07 dist-test-slave-x4qh krb5kdc[6126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:51:07.974529 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:46217
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:46217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:08.049765 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:08.049937 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:08.049957 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:08.051043 6144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:51:08.051082 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:08.051097 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:08.051110 6144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:51:08.051156 6144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:51:08.052503 6144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:46217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:08.052655 6144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:08.052851 6144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:08.057217 6154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.057219 6150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.057265 6151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:08.057619 6144 server_base.cc:1034] running on GCE node
I20260808 22:51:08.057881 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:08.058156 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:08.059291 6144 hybrid_clock.cc:648] HybridClock initialized: now 1786229468059272 us; error 34 us; skew 500 ppm
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:08.061884 6144 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:51:08.062341 6144 webserver.cc:460] Webserver started at http://127.22.30.62:39815/ using document root <none> and password file <none>
I20260808 22:51:08.062506 6144 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:08.062543 6144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:08.062623 6144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:08.063545 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "b315c8fe980849ccbb97785e7200be15"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "df51b07e1a8f5a2e168108f1f616b69b"
server_key_iv: "06c81c8f6ecd6e8ba0707e2508021bb3"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.063812 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "b315c8fe980849ccbb97785e7200be15"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "df51b07e1a8f5a2e168108f1f616b69b"
server_key_iv: "06c81c8f6ecd6e8ba0707e2508021bb3"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.064977 6144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:51:08.065605 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.065768 6144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:08.065861 6144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "b315c8fe980849ccbb97785e7200be15"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "df51b07e1a8f5a2e168108f1f616b69b"
server_key_iv: "06c81c8f6ecd6e8ba0707e2508021bb3"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.065937 6144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:08.092625 6144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:08.094640 6144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:08.094772 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:08.098488 6144 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:46217
I20260808 22:51:08.098516 6239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:46217 every 8 connection(s)
I20260808 22:51:08.098821 6144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:51:08.099699 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:08.101610 6240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15: Bootstrap starting.
I20260808 22:51:08.102097 6240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:08.102312 6240 log.cc:824] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:08.102883 6240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15: No bootstrap required, opened a new log
I20260808 22:51:08.104255 6240 raft_consensus.cc:348] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b315c8fe980849ccbb97785e7200be15" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46217 } }
I20260808 22:51:08.104382 6240 raft_consensus.cc:374] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:08.104422 6240 raft_consensus.cc:729] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b315c8fe980849ccbb97785e7200be15, State: Initialized, Role: FOLLOWER
I20260808 22:51:08.104512 6240 consensus_queue.cc:260] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b315c8fe980849ccbb97785e7200be15" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46217 } }
I20260808 22:51:08.104560 6240 raft_consensus.cc:388] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:51:08.104579 6240 raft_consensus.cc:482] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:51:08.104647 6240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:08.105389 6240 raft_consensus.cc:504] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b315c8fe980849ccbb97785e7200be15" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46217 } }
I20260808 22:51:08.105497 6240 leader_election.cc:302] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b315c8fe980849ccbb97785e7200be15; no voters:
I20260808 22:51:08.105677 6240 leader_election.cc:288] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:51:08.105713 6244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:51:08.105844 6244 raft_consensus.cc:686] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [term 1 LEADER]: Becoming Leader. State: Replica: b315c8fe980849ccbb97785e7200be15, State: Running, Role: LEADER
I20260808 22:51:08.105916 6240 sys_catalog.cc:564] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:51:08.105978 6244 consensus_queue.cc:237] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b315c8fe980849ccbb97785e7200be15" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46217 } }
I20260808 22:51:08.106312 6245 sys_catalog.cc:455] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b315c8fe980849ccbb97785e7200be15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b315c8fe980849ccbb97785e7200be15" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46217 } } }
I20260808 22:51:08.106415 6245 sys_catalog.cc:458] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:08.106313 6246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b315c8fe980849ccbb97785e7200be15. Latest consensus state: current_term: 1 leader_uuid: "b315c8fe980849ccbb97785e7200be15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b315c8fe980849ccbb97785e7200be15" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46217 } } }
I20260808 22:51:08.106527 6246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15 [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:08.106607 6257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:51:08.107038 6257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:51:08.108402 6257 catalog_manager.cc:1347] Generated new cluster ID: 5715285a8cc54a388529b78ea438223d
I20260808 22:51:08.108453 6257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:51:08.108530 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 6144
I20260808 22:51:08.108660 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:51:08.108837 22648 external_mini_cluster.cc:1428] Setting key f57b9a5430a570043cab22dbdc3c9cb1
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:51:08.120124 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.109441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.22.30.62:46217)
0808 22:51:08.111566 (+ 2125us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.111572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.111594 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:51:08.111672 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.111677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.111712 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.111846 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:08.112414 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.113242 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.113974 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.114364 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.114513 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.116754 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:08.116758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:08.116758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:08.116769 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:08.118453 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.118886 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.118891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.118891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.118971 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.119177 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.119178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.119179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.119462 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:08.119593 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.119727 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.119972 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2096,"thread_start_us":73,"threads_started":1}
I20260808 22:51:08.135015 6257 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:51:08.135548 6257 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:51:08.144577 6257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15: Generated new TSK 0
I20260808 22:51:08.144732 6257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:51:08.176286 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:46217
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:08.254210 6279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:08.254376 6279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:08.254396 6279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:08.255352 6279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:08.255406 6279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:08.255435 6279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:51:08.256709 6279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:46217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:08.256876 6279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:08.257028 6279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:08.261524 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.261515 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.261516 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:08.261758 6279 server_base.cc:1034] running on GCE node
I20260808 22:51:08.262041 6279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:08.262275 6279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:08.263432 6279 hybrid_clock.cc:648] HybridClock initialized: now 1786229468263421 us; error 24 us; skew 500 ppm
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:08.266036 6279 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:51:08.266528 6279 webserver.cc:460] Webserver started at http://127.22.30.1:41729/ using document root <none> and password file <none>
I20260808 22:51:08.266745 6279 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:08.266811 6279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:08.266930 6279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:08.268007 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "c72dd9a1412d4eb9aebec9948d1464a0"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "9ad7f34fb6cd58f791f452319c46a094"
server_key_iv: "de25edd06035e912c2bad192c96c8869"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.268395 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "c72dd9a1412d4eb9aebec9948d1464a0"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "9ad7f34fb6cd58f791f452319c46a094"
server_key_iv: "de25edd06035e912c2bad192c96c8869"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.269830 6279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:51:08.270599 6303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.270784 6279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:08.270865 6279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "c72dd9a1412d4eb9aebec9948d1464a0"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "9ad7f34fb6cd58f791f452319c46a094"
server_key_iv: "de25edd06035e912c2bad192c96c8869"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.270916 6279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:08.282931 6279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:08.284634 6279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:08.284726 6279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:08.284871 6279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:08.285074 6279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:08.285111 6279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.285130 6279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:08.285141 6279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.290161 6279 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:33079
I20260808 22:51:08.290282 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:33079 every 8 connection(s)
I20260808 22:51:08.290558 6279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:51:08.298985 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.291218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:33541 (local address 127.22.30.62:46217)
0808 22:51:08.291291 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.291293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.291459 (+ 166us) server_negotiation.cc:407] Connection header received
0808 22:51:08.291888 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.291889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.291902 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.291952 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:08.292779 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.293244 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.294132 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.294357 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.294408 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.296277 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:08.296278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:08.296278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:08.296283 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:08.297461 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.297966 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.297968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.297968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.298063 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.298374 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.298375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.298375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.298510 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:08.298571 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.298784 (+ 213us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.298887 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:51:08.299103 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.291165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:46217 (local address 127.22.30.1:33541)
0808 22:51:08.291315 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.291346 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.291804 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.291997 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.292001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.292507 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:08.292700 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.292705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.293319 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.293320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.294057 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.294059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.294430 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.294431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.294526 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.294921 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:08.294929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:08.296195 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:08.297569 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:08.297576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:08.297578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:08.297873 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:08.298132 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:08.298133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:08.298133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:08.298281 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:08.298562 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:08.298566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:08.298674 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.298745 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.298892 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260808 22:51:08.299727 6477 heartbeater.cc:347] Connected to a master server at 127.22.30.62:46217
I20260808 22:51:08.299839 6477 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:08.299997 6477 heartbeater.cc:511] Master 127.22.30.62:46217 requested a full tablet report, sending...
I20260808 22:51:08.300545 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 6279
I20260808 22:51:08.300659 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:51:08.300858 22648 external_mini_cluster.cc:1428] Setting key b0fdd9659ce772ddbbde781bb66c8abe
I20260808 22:51:08.300884 6193 ts_manager.cc:194] Registered new tserver with Master: c72dd9a1412d4eb9aebec9948d1464a0 (127.22.30.1:33079)
I20260808 22:51:08.301539 6193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:33541
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:51:08.323781 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:46217
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:08.398813 6487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:08.398975 6487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:08.398995 6487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:08.400079 6487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:08.400120 6487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:08.400146 6487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:51:08.401602 6487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:46217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:08.401755 6487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:08.401964 6487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:08.406787 6497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.406809 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.406790 6499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:08.406991 6487 server_base.cc:1034] running on GCE node
I20260808 22:51:08.407177 6487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:08.407341 6487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:08.408522 6487 hybrid_clock.cc:648] HybridClock initialized: now 1786229468408516 us; error 39 us; skew 500 ppm
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:08.410835 6487 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:51:08.411286 6487 webserver.cc:460] Webserver started at http://127.22.30.2:40725/ using document root <none> and password file <none>
I20260808 22:51:08.411538 6487 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:08.411583 6487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:08.411793 6487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:08.412746 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "67bbd0b1fb094b01906629bd34758f96"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "d7ab15a1f37617a50ea6f59a62197210"
server_key_iv: "5ad258308b2d5876dd45f96ebf46dc2a"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.413089 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "67bbd0b1fb094b01906629bd34758f96"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "d7ab15a1f37617a50ea6f59a62197210"
server_key_iv: "5ad258308b2d5876dd45f96ebf46dc2a"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.414242 6487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:51:08.414909 6513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.415086 6487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:51:08.415155 6487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "67bbd0b1fb094b01906629bd34758f96"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "d7ab15a1f37617a50ea6f59a62197210"
server_key_iv: "5ad258308b2d5876dd45f96ebf46dc2a"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.415238 6487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:08.429337 6487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:08.431464 6487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:08.431569 6487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:08.431715 6487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:08.431919 6487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:08.431947 6487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.431964 6487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:08.431977 6487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.437438 6487 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:39293
I20260808 22:51:08.437479 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:39293 every 8 connection(s)
I20260808 22:51:08.437747 6487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:51:08.438047 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 6487
I20260808 22:51:08.438139 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:51:08.438349 22648 external_mini_cluster.cc:1428] Setting key fd813f8bd95c3d8f248cdfb04833583a
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
I20260808 22:51:08.447151 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.438396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:45613 (local address 127.22.30.62:46217)
0808 22:51:08.438458 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.438460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.438705 (+ 245us) server_negotiation.cc:407] Connection header received
0808 22:51:08.439261 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.439261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.439270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.439307 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:08.440497 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.440998 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.441938 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.442125 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.442196 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.444117 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:08.444118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:08.444118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:08.444126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:08.445730 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.446259 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.446260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.446260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.446335 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.446565 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.446565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.446566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.446674 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:08.446747 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.446985 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.447060 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:51:08.447296 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.438326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:46217 (local address 127.22.30.2:45613)
0808 22:51:08.438534 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.438612 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.439159 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.439391 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.439396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.440168 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:08.440410 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.440416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.441091 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.441093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.441849 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.441851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.442204 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.442205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.442328 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.442756 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:08.442763 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:08.444025 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:08.445820 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:08.445826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:08.445828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:08.446170 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:08.446400 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:08.446401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:08.446401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:08.446501 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:08.446738 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:08.446744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:08.446868 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.446965 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.447098 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20260808 22:51:08.447773 6679 heartbeater.cc:347] Connected to a master server at 127.22.30.62:46217
I20260808 22:51:08.447862 6679 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:08.448045 6679 heartbeater.cc:511] Master 127.22.30.62:46217 requested a full tablet report, sending...
I20260808 22:51:08.448388 6193 ts_manager.cc:194] Registered new tserver with Master: 67bbd0b1fb094b01906629bd34758f96 (127.22.30.2:39293)
I20260808 22:51:08.448722 6193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:45613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:51:08.461418 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:46217
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:08.535568 6686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:08.535728 6686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:08.535750 6686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:08.536752 6686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:08.536787 6686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:08.536809 6686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:51:08.537994 6686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:46217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:08.538143 6686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:08.538292 6686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:08.542713 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.542708 6691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:08.542709 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:08.543061 6686 server_base.cc:1034] running on GCE node
I20260808 22:51:08.543267 6686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:08.543521 6686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:08.544669 6686 hybrid_clock.cc:648] HybridClock initialized: now 1786229468544664 us; error 34 us; skew 500 ppm
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:08.546857 6686 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:51:08.547215 6686 webserver.cc:460] Webserver started at http://127.22.30.3:41855/ using document root <none> and password file <none>
I20260808 22:51:08.547492 6686 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:08.547564 6686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:08.547698 6686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:51:08.548590 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "85d7f92c5e3c41258fc2eed9d601520b"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "e00cd9da81b0fb958b0d6e30b5830cd9"
server_key_iv: "8e89f24d3d782fa15204322b87d5723e"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.548915 6686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "85d7f92c5e3c41258fc2eed9d601520b"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "e00cd9da81b0fb958b0d6e30b5830cd9"
server_key_iv: "8e89f24d3d782fa15204322b87d5723e"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.550122 6686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:51:08.550802 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.550966 6686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:51:08.551059 6686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "85d7f92c5e3c41258fc2eed9d601520b"
format_stamp: "Formatted at 2026-08-08 22:51:08 on dist-test-slave-x4qh"
server_key: "e00cd9da81b0fb958b0d6e30b5830cd9"
server_key_iv: "8e89f24d3d782fa15204322b87d5723e"
server_key_version: "encryptionkey@0"
I20260808 22:51:08.551141 6686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:08.568040 6686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:08.570060 6686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:08.570221 6686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:08.570454 6686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:08.570735 6686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:08.570791 6686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.570825 6686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:08.570851 6686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:08.576596 6686 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:36829
I20260808 22:51:08.576642 6876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:36829 every 8 connection(s)
I20260808 22:51:08.576924 6686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:51:08.585399 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 6686
I20260808 22:51:08.585508 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:51:08.585687 22648 external_mini_cluster.cc:1428] Setting key ca26f3f0ab9ad1bfa127441a9fa926f3
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229468, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:51:08.588919 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.577547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:58375 (local address 127.22.30.62:46217)
0808 22:51:08.577602 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.577604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.577766 (+ 162us) server_negotiation.cc:407] Connection header received
0808 22:51:08.578122 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.578123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.578132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.578182 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:08.578917 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.579358 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.580217 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.580436 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.580485 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.586829 (+ 6344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:08.586830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:08.586830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:08.586836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:08.587948 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.588258 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.588259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.588259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.588305 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.588466 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.588467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.588467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.588549 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:08.588592 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.588810 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.588864 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:51:08.589038 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.577516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:46217 (local address 127.22.30.3:58375)
0808 22:51:08.577661 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.577692 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.578040 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.578220 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.578223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.578689 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:08.578849 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.578853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.579482 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.579483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.580150 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.580151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.580505 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.580506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.580594 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.585289 (+ 4695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:08.585296 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:08.586751 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:08.588013 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:08.588018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:08.588019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:08.588198 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:08.588352 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:08.588352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:08.588352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:08.588422 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:08.588599 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:08.588603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:08.588709 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.588787 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.588884 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260808 22:51:08.589488 6878 heartbeater.cc:347] Connected to a master server at 127.22.30.62:46217
I20260808 22:51:08.589592 6878 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:08.589723 6878 heartbeater.cc:511] Master 127.22.30.62:46217 requested a full tablet report, sending...
I20260808 22:51:08.590056 6193 ts_manager.cc:194] Registered new tserver with Master: 85d7f92c5e3c41258fc2eed9d601520b (127.22.30.3:36829)
I20260808 22:51:08.590368 6193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:58375
I20260808 22:51:08.597468 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:51:08.603896 6275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.598571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.22.30.62:46217)
0808 22:51:08.598625 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.598626 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.598746 (+ 120us) server_negotiation.cc:407] Connection header received
0808 22:51:08.598844 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.598845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.598854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.598909 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:08.599478 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.599884 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.600603 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.600797 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.600867 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.601846 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:08.601847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:08.601847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:08.601854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:08.602813 (+ 959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.603127 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.603129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.603129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.603179 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:08.603395 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:08.603396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:08.603396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:08.603516 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:08.603644 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.603746 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.603812 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:51:08.605122 6193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:51:08.606726 6193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:51:08.612380 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.608786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36829 (local address 127.0.0.1:52662)
0808 22:51:08.608907 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.608927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.609004 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.609213 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.609216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.609217 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:08.609273 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.609275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.610235 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.610236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.611412 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.611415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.612094 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.612095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.612186 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.612202 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.612270 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.612318 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":44,"threads_started":1}
I20260808 22:51:08.612401 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.608876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.22.30.3:36829)
0808 22:51:08.609047 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.609051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.609060 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:51:08.609092 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.609094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.609106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.609175 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:08.609495 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.610141 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.611505 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.612013 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.612093 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.612141 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.612303 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.612335 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260808 22:51:08.612701 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.608564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33079 (local address 127.0.0.1:42714)
0808 22:51:08.608893 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.608927 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.609004 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.609429 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.609430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.609432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:08.609562 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.609564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.610698 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.610699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.611576 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.611578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.612458 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.612459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.612536 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.612540 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.612593 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.612637 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":51,"threads_started":1}
I20260808 22:51:08.612751 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.608680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.22.30.1:33079)
0808 22:51:08.609275 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.609278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.609284 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:51:08.609337 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.609339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.609352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.609420 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:08.609617 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.610618 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.611640 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.612307 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.612375 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.612422 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.612628 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.612672 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":127,"threads_started":1}
I20260808 22:51:08.613090 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 725389497eae4e269c8896a90a4e086a (DEFAULT_TABLE table=test-table [id=c53a4882bc1c4485b935d6ead69a51d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:51:08.613237 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.608690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.22.30.2:39293)
0808 22:51:08.609137 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.609141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.609172 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:51:08.609220 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.609222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.609235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.609322 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:08.609463 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.610488 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.611688 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.612310 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.612375 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.612533 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.612738 (+ 205us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.613127 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":129,"threads_started":1}
I20260808 22:51:08.613387 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.608583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:39293 (local address 127.0.0.1:53106)
0808 22:51:08.608909 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.608955 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.609009 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.609334 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.609335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.609337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:08.609382 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.609383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.610578 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.610579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.611629 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.611631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.612561 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.612562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.612614 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.612616 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.612662 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.612689 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":55,"threads_started":1}
I20260808 22:51:08.613482 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725389497eae4e269c8896a90a4e086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:08.613413 6378 tablet_service.cc:1467] Processing CreateTablet for tablet 725389497eae4e269c8896a90a4e086a (DEFAULT_TABLE table=test-table [id=c53a4882bc1c4485b935d6ead69a51d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:51:08.613792 6378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725389497eae4e269c8896a90a4e086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:08.615948 6901 tablet_bootstrap.cc:492] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Bootstrap starting.
I20260808 22:51:08.615967 6579 tablet_service.cc:1467] Processing CreateTablet for tablet 725389497eae4e269c8896a90a4e086a (DEFAULT_TABLE table=test-table [id=c53a4882bc1c4485b935d6ead69a51d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:51:08.616290 6579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725389497eae4e269c8896a90a4e086a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:08.616330 6902 tablet_bootstrap.cc:492] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0: Bootstrap starting.
I20260808 22:51:08.616582 6901 tablet_bootstrap.cc:654] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:08.616990 6902 tablet_bootstrap.cc:654] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:08.617130 6901 log.cc:824] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:08.617550 6902 log.cc:824] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:08.617939 6901 tablet_bootstrap.cc:492] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: No bootstrap required, opened a new log
I20260808 22:51:08.618007 6901 ts_tablet_manager.cc:1397] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:51:08.618259 6902 tablet_bootstrap.cc:492] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0: No bootstrap required, opened a new log
I20260808 22:51:08.618350 6902 ts_tablet_manager.cc:1397] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:51:08.619536 6907 tablet_bootstrap.cc:492] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Bootstrap starting.
I20260808 22:51:08.619591 6901 raft_consensus.cc:348] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.619733 6901 raft_consensus.cc:374] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:08.619772 6901 raft_consensus.cc:729] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d7f92c5e3c41258fc2eed9d601520b, State: Initialized, Role: FOLLOWER
I20260808 22:51:08.619884 6901 consensus_queue.cc:260] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.620180 6878 heartbeater.cc:503] Master 127.22.30.62:46217 was elected leader, sending a full tablet report...
I20260808 22:51:08.620420 6907 tablet_bootstrap.cc:654] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:08.620172 6901 ts_tablet_manager.cc:1428] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:51:08.620551 6902 raft_consensus.cc:348] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.620687 6902 raft_consensus.cc:374] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:08.620718 6902 raft_consensus.cc:729] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72dd9a1412d4eb9aebec9948d1464a0, State: Initialized, Role: FOLLOWER
I20260808 22:51:08.620837 6902 consensus_queue.cc:260] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.621003 6907 log.cc:824] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:08.621114 6902 ts_tablet_manager.cc:1428] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:51:08.621150 6477 heartbeater.cc:503] Master 127.22.30.62:46217 was elected leader, sending a full tablet report...
I20260808 22:51:08.621980 6907 tablet_bootstrap.cc:492] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: No bootstrap required, opened a new log
I20260808 22:51:08.622051 6907 ts_tablet_manager.cc:1397] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:51:08.623311 6907 raft_consensus.cc:348] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.623485 6907 raft_consensus.cc:374] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:08.623512 6907 raft_consensus.cc:729] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bbd0b1fb094b01906629bd34758f96, State: Initialized, Role: FOLLOWER
I20260808 22:51:08.623627 6907 consensus_queue.cc:260] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.625403 6907 ts_tablet_manager.cc:1428] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:51:08.625473 6679 heartbeater.cc:503] Master 127.22.30.62:46217 was elected leader, sending a full tablet report...
W20260808 22:51:08.688380 6680 tablet.cc:2369] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:51:08.717509 6911 raft_consensus.cc:482] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:51:08.717592 6911 raft_consensus.cc:504] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.717922 6911 leader_election.cc:288] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c72dd9a1412d4eb9aebec9948d1464a0 (127.22.30.1:33079), 85d7f92c5e3c41258fc2eed9d601520b (127.22.30.3:36829)
I20260808 22:51:08.721751 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.718070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33079 (local address 127.22.30.2:45797)
0808 22:51:08.718146 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.718156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.718197 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.718377 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.718378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.718379 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:08.718469 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.718471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.719399 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.719400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.720675 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.720678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.721469 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.721470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.721556 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.721561 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.721617 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.721663 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":68}
I20260808 22:51:08.721752 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.718117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:45797 (local address 127.22.30.1:33079)
0808 22:51:08.718154 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.718156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.718185 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:51:08.718264 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.718264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.718282 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.718358 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:08.718569 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.719308 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.720764 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.721371 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.721456 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.721466 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.721647 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.721687 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:51:08.721752 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.718137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57643 (local address 127.22.30.3:36829)
0808 22:51:08.718187 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.718189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.718318 (+ 129us) server_negotiation.cc:407] Connection header received
0808 22:51:08.718345 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.718346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.718358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.718406 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:08.718572 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.719298 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.720765 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.721367 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.721456 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.721465 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.721647 (+ 182us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.721687 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 22:51:08.721776 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.718070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36829 (local address 127.22.30.2:57643)
0808 22:51:08.718250 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.718258 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:08.718312 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:08.718407 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:08.718408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:08.718409 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:08.718485 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.718486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.719399 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.719400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.720675 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:08.720678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.721470 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.721470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:08.721556 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.721561 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:51:08.721617 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:51:08.721662 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":134,"thread_start_us":101,"threads_started":1}
I20260808 22:51:08.722165 6405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725389497eae4e269c8896a90a4e086a" candidate_uuid: "67bbd0b1fb094b01906629bd34758f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" is_pre_election: true
I20260808 22:51:08.722319 6405 raft_consensus.cc:2454] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bbd0b1fb094b01906629bd34758f96 in term 0.
I20260808 22:51:08.722402 6809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725389497eae4e269c8896a90a4e086a" candidate_uuid: "67bbd0b1fb094b01906629bd34758f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d7f92c5e3c41258fc2eed9d601520b" is_pre_election: true
I20260808 22:51:08.722541 6809 raft_consensus.cc:2454] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bbd0b1fb094b01906629bd34758f96 in term 0.
I20260808 22:51:08.722553 6514 leader_election.cc:302] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bbd0b1fb094b01906629bd34758f96, c72dd9a1412d4eb9aebec9948d1464a0; no voters:
I20260808 22:51:08.722674 6911 raft_consensus.cc:2781] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:51:08.722738 6911 raft_consensus.cc:482] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:51:08.722771 6911 raft_consensus.cc:3037] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:08.723510 6911 raft_consensus.cc:504] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.723641 6911 leader_election.cc:288] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [CANDIDATE]: Term 1 election: Requested vote from peers c72dd9a1412d4eb9aebec9948d1464a0 (127.22.30.1:33079), 85d7f92c5e3c41258fc2eed9d601520b (127.22.30.3:36829)
I20260808 22:51:08.723788 6405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725389497eae4e269c8896a90a4e086a" candidate_uuid: "67bbd0b1fb094b01906629bd34758f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72dd9a1412d4eb9aebec9948d1464a0"
I20260808 22:51:08.723830 6809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725389497eae4e269c8896a90a4e086a" candidate_uuid: "67bbd0b1fb094b01906629bd34758f96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d7f92c5e3c41258fc2eed9d601520b"
I20260808 22:51:08.723866 6405 raft_consensus.cc:3037] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:08.723901 6809 raft_consensus.cc:3037] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:08.724576 6405 raft_consensus.cc:2454] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bbd0b1fb094b01906629bd34758f96 in term 1.
I20260808 22:51:08.724651 6809 raft_consensus.cc:2454] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bbd0b1fb094b01906629bd34758f96 in term 1.
I20260808 22:51:08.724826 6515 leader_election.cc:302] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bbd0b1fb094b01906629bd34758f96, 85d7f92c5e3c41258fc2eed9d601520b; no voters:
I20260808 22:51:08.724933 6911 raft_consensus.cc:2781] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:51:08.725085 6911 raft_consensus.cc:686] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 1 LEADER]: Becoming Leader. State: Replica: 67bbd0b1fb094b01906629bd34758f96, State: Running, Role: LEADER
I20260808 22:51:08.725174 6911 consensus_queue.cc:237] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } }
I20260808 22:51:08.725836 6192 catalog_manager.cc:5495] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67bbd0b1fb094b01906629bd34758f96 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "67bbd0b1fb094b01906629bd34758f96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67bbd0b1fb094b01906629bd34758f96" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 39293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 } health_report { overall_health: UNKNOWN } } }
I20260808 22:51:08.731033 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.728925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.22.30.2:39293)
0808 22:51:08.728963 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:08.728965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:08.728973 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:51:08.729032 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:08.729032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:08.729049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:08.729124 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:51:08.729233 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.729604 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.730259 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:08.730473 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:08.730555 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:08.730726 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:51:08.730816 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:51:08.730903 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:08.730931 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 22:51:08.730947 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:51:08.732506 6405 raft_consensus.cc:1261] T 725389497eae4e269c8896a90a4e086a P c72dd9a1412d4eb9aebec9948d1464a0 [term 1 FOLLOWER]: Refusing update from remote peer 67bbd0b1fb094b01906629bd34758f96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:51:08.732506 6809 raft_consensus.cc:1261] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 1 FOLLOWER]: Refusing update from remote peer 67bbd0b1fb094b01906629bd34758f96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:51:08.732826 6913 consensus_queue.cc:1035] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72dd9a1412d4eb9aebec9948d1464a0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:51:08.732935 6911 consensus_queue.cc:1035] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d7f92c5e3c41258fc2eed9d601520b" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:51:08.735925 6917 mvcc.cc:205] Tried to move back new op lower bound from 7316395903926333440 to 7316395903898861568. Current Snapshot: MvccSnapshot[applied={T|T < 7316395903926333440}]
I20260808 22:51:08.736208 6923 mvcc.cc:205] Tried to move back new op lower bound from 7316395903926333440 to 7316395903898861568. Current Snapshot: MvccSnapshot[applied={T|T < 7316395903926333440 or (T in {7316395903926333440})}]
I20260808 22:51:08.736325 6924 mvcc.cc:205] Tried to move back new op lower bound from 7316395903926333440 to 7316395903898861568. Current Snapshot: MvccSnapshot[applied={T|T < 7316395903926333440}]
I20260808 22:51:08.738101 6192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:51:08.738181 6192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:51:08.738833 6192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b315c8fe980849ccbb97785e7200be15: Sending DeleteTablet for 3 replicas of tablet 725389497eae4e269c8896a90a4e086a
I20260808 22:51:08.739143 6378 tablet_service.cc:1514] Processing DeleteTablet for tablet 725389497eae4e269c8896a90a4e086a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:51:08 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:42714
I20260808 22:51:08.739169 6579 tablet_service.cc:1514] Processing DeleteTablet for tablet 725389497eae4e269c8896a90a4e086a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:51:08 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:53106
I20260808 22:51:08.739166 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 725389497eae4e269c8896a90a4e086a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:51:08 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:52662
I20260808 22:51:08.739434 6941 tablet_replica.cc:330] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: stopping tablet replica
I20260808 22:51:08.739519 6941 raft_consensus.cc:2229] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:51:08.739607 6941 raft_consensus.cc:2258] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:51:08.739648 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 6279
I20260808 22:51:08.743600 6941 ts_tablet_manager.cc:1905] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:51:08.745028 6940 tablet_replica.cc:330] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: stopping tablet replica
I20260808 22:51:08.745151 6941 ts_tablet_manager.cc:1918] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:51:08.745260 6941 log.cc:1196] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/725389497eae4e269c8896a90a4e086a
I20260808 22:51:08.745659 6941 ts_tablet_manager.cc:1939] T 725389497eae4e269c8896a90a4e086a P 85d7f92c5e3c41258fc2eed9d601520b: Deleting consensus metadata
I20260808 22:51:08.745697 6940 raft_consensus.cc:2229] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:51:08.745878 6940 raft_consensus.cc:2258] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:51:08.748665 6166 catalog_manager.cc:4841] TS 85d7f92c5e3c41258fc2eed9d601520b (127.22.30.3:36829): tablet 725389497eae4e269c8896a90a4e086a (table test-table [id=c53a4882bc1c4485b935d6ead69a51d2]) successfully deleted
I20260808 22:51:08.748798 6940 ts_tablet_manager.cc:1905] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:51:08.750006 6940 ts_tablet_manager.cc:1918] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:51:08.750064 6940 log.cc:1196] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/725389497eae4e269c8896a90a4e086a
I20260808 22:51:08.750263 6940 ts_tablet_manager.cc:1939] T 725389497eae4e269c8896a90a4e086a P 67bbd0b1fb094b01906629bd34758f96: Deleting consensus metadata
I20260808 22:51:08.750595 6166 catalog_manager.cc:4841] TS 67bbd0b1fb094b01906629bd34758f96 (127.22.30.2:39293): tablet 725389497eae4e269c8896a90a4e086a (table test-table [id=c53a4882bc1c4485b935d6ead69a51d2]) successfully deleted
W20260808 22:51:08.754422 6165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:33079 (error 108)
I20260808 22:51:08.754691 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 6487
I20260808 22:51:08.757409 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:08.754800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:33079 (local address 127.0.0.1:42730)
0808 22:51:08.757328 (+ 2528us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:08.757373 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2504}
W20260808 22:51:08.757522 6165 catalog_manager.cc:4584] TS c72dd9a1412d4eb9aebec9948d1464a0 (127.22.30.1:33079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 725389497eae4e269c8896a90a4e086a: Network error: Client connection negotiation failed: client connection to 127.22.30.1:33079: connect: Connection refused (error 111)
I20260808 22:51:08.763222 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 6686
I20260808 22:51:08.770223 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 6144
2026-08-08T22:51:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2069 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6945](info): set up 2 sockets
Aug 08 22:51:08 dist-test-slave-x4qh krb5kdc[6945](info): commencing operation
krb5kdc: starting...
W20260808 22:51:09.956194 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:51:10 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:51:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:51:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/rangeradmin_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/rangeradmin_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/rangerlookup_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/rangerlookup_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
I20260808 22:51:10.106066 22648 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskT1Dm48/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/postgres -l logfile start
2026-08-08 22:51:12.193 UTC [6998] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 22:51:12.194 UTC [6998] LOG: listening on IPv4 address "127.22.30.20", port 32993
2026-08-08 22:51:12.196 UTC [6998] LOG: listening on Unix socket "/tmp/.s.PGSQL.32993"
2026-08-08 22:51:12.216 UTC [7003] LOG: database system was shut down at 2026-08-08 22:51:11 UTC
2026-08-08 22:51:12.220 UTC [6998] LOG: database system is ready to accept connections
I20260808 22:51:13.310387 22648 mini_postgres.cc:96] Postgres bound to 32993
2026-08-08 22:51:13.317 UTC [7018] FATAL: database "slave" does not exist
127.22.30.20:32993 - accepting connections
I20260808 22:51:13.318552 22648 mini_ranger.cc:161] Starting Ranger...
I20260808 22:51:13.332651 22648 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 22:51:13.704731 22648 mini_ranger.cc:87] Created ranger Postgres database
I20260808 22:51:13.704831 22648 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-admin
2026-08-08 22:51:13,967 [I] DB FLAVOR :POSTGRES
2026-08-08 22:51:13,968 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:51:13,968 [I] Checking connection..
2026-08-08 22:51:13,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 22:51:14,214 [I] Checking connection passed.
2026-08-08 22:51:14,214 [I] --------- Verifying version history table ---------
2026-08-08 22:51:14,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:51:14,467 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:51:14,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:51:14,723 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:51:14,724 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 22:51:14,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 22:51:14.949 UTC [7114] WARNING: there is no transaction in progress
2026-08-08 22:51:14,956 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 22:51:14,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:51:15,193 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 22:51:15,194 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:51:15,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 22:51:15,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 22:51:15,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x4qh.c.gcp-upstream.internal','N') ;"
2026-08-08 22:51:15,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:51:16,137 [I] Table x_portal_user does not exist in database ranger
2026-08-08 22:51:16,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:51:16,384 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 22:51:16,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:51:16,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:51:16,831 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 22:51:16,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 22:51:17.655 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.668 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.682 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.695 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.708 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.723 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.770 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.776 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.787 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.795 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.804 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.816 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.824 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.830 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.835 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:17.840 UTC [7284] WARNING: there is no transaction in progress
2026-08-08 22:51:18,399 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 22:51:18,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 22:51:18,642 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 22:51:18,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:51:18,900 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 22:51:18,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:51:19,145 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 22:51:19,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:51:19,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:51:19,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x4qh.c.gcp-upstream.internal';"
2026-08-08 22:51:19,843 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 22:51:19,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 22:51:20,093 [I] DB_PATCHES have already been applied
I20260808 22:51:20.098667 22648 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-admin:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskT1Dm48/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 22:51:20.098724 22648 mini_ranger.cc:193] Using host: 127.22.30.20
I20260808 22:51:20.102771 22648 mini_ranger.cc:227] Ranger admin URL: http://127.22.30.20:35251
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 10:51:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 22:51:21.389214 22648 mini_ranger.cc:160] Time spent starting Ranger: real 8.071s user 0.000s sys 0.010s
Aug 08, 2026 10:51:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 10:51:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 08, 2026 10:51:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 10:51:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 10:51:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 22:51:41.203697 22648 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/rangerkms_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/rangerkms_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.30.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.30.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 22:51:41 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229501, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 22:51:41.276031 22648 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 22:51:41.284947 22648 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 22:51:41.989949 22648 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 22:51:41.990042 22648 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms
I20260808 22:51:41.990059 22648 mini_ranger_kms.cc:227] Using postgres at 127.22.30.20:32993
2026-08-08 22:51:42,251 [I] DB FLAVOR :POSTGRES
2026-08-08 22:51:42,252 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:51:42,252 [I] Checking connection
2026-08-08 22:51:42,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 22:51:42,517 [I] connection success
2026-08-08 22:51:42,517 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 22:51:42,517 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 22:51:42,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 22:51:42,736 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 22:51:42,737 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:51:42,737 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 22:51:42,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:32993/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 22:51:42,983 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 22:51:43.399319 22648 mini_ranger_kms.cc:310] Created kms service
I20260808 22:51:43.593302 22648 mini_ranger_kms.cc:326] Created kudu user
I20260808 22:51:43.634194 22648 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 22:51:43 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229501, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:51:44.016695 22648 mini_ranger_kms.cc:384] Added ranger policy
I20260808 22:51:44.016821 22648 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms:/tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 22:51:44.016844 22648 mini_ranger_kms.cc:240] Using host: 127.22.30.20
I20260808 22:51:44.021705 22648 mini_ranger_kms.cc:276] Ranger KMS PID: 7571
I20260808 22:51:44.021831 22648 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.30.20:33149
Aug 08, 2026 10:51:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:51:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:51:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 10:51:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 10:51:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 10:51:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 22:51:46 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786229506, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.30.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:46.700685 22648 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.425s user 0.000s sys 0.244s
I20260808 22:51:46.700783 22648 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 22:51:46.700872 22648 mini_ranger_kms.cc:401] 127.22.30.20:33149/kms/v1/keys
Aug 08 22:51:46 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229506, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.30.20@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
Aug 08 22:51:47 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229506, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.30.20@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
Aug 08 22:51:49 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:51:49.726002 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42475
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:42475
--ranger_config_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:33149/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:49.845454 7655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:49.845669 7655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:49.845705 7655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:49.846815 7655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:51:49.846889 7655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:49.846915 7655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:49.846959 7655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:51:49.847012 7655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:51:49.848729 7655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.30.20:33149/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:42475
--ranger_config_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:49.848995 7655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:49.849233 7655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:49.854281 7663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:49.854306 7664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:49.854382 7655 server_base.cc:1034] running on GCE node
W20260808 22:51:49.854547 7666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:49.854801 7655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:49.855134 7655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:49.856297 7655 hybrid_clock.cc:648] HybridClock initialized: now 1786229509856276 us; error 38 us; skew 500 ppm
Aug 08 22:51:49 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229509, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:49.864070 7655 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:51:49.864552 7655 webserver.cc:460] Webserver started at http://127.22.30.62:38881/ using document root <none> and password file <none>
I20260808 22:51:49.864800 7655 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:49.864863 7655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:49.865029 7655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:51:49 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229509, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:51:49.924695 7655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "e3b2061b278b4a98b5d3bf841ee91ad6"
format_stamp: "Formatted at 2026-08-08 22:51:49 on dist-test-slave-x4qh"
server_key: "4630d45df73645b10d409421f544b221"
server_key_iv: "4d1a7824270ebbe793f45f584e3119e9"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:49.925086 7655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "e3b2061b278b4a98b5d3bf841ee91ad6"
format_stamp: "Formatted at 2026-08-08 22:51:49 on dist-test-slave-x4qh"
server_key: "4630d45df73645b10d409421f544b221"
server_key_iv: "4d1a7824270ebbe793f45f584e3119e9"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:49.926813 7655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:51:49.940455 7676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:49.940642 7655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:51:49.940728 7655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "e3b2061b278b4a98b5d3bf841ee91ad6"
format_stamp: "Formatted at 2026-08-08 22:51:49 on dist-test-slave-x4qh"
server_key: "4630d45df73645b10d409421f544b221"
server_key_iv: "4d1a7824270ebbe793f45f584e3119e9"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:49.940816 7655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:49.957376 7655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:49.959326 7655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:49.959518 7655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:49.969669 7655 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:42475
I20260808 22:51:49.969785 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:42475 every 8 connection(s)
I20260808 22:51:49.970151 7655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:51:49.971083 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:49.972712 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 7655
I20260808 22:51:49.972865 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:51:49.973140 22648 external_mini_cluster.cc:1428] Setting key 6c1afe77dd1c6f9b276abe0bdf6e980b
I20260808 22:51:49.974996 7759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6: Bootstrap starting.
I20260808 22:51:49.976310 7759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:49.977005 7759 log.cc:824] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:49.977787 7759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6: No bootstrap required, opened a new log
I20260808 22:51:49.981622 7759 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42475 } }
I20260808 22:51:49.981788 7759 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:49.981828 7759 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b2061b278b4a98b5d3bf841ee91ad6, State: Initialized, Role: FOLLOWER
Aug 08 22:51:49 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:51:49.982551 7759 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42475 } }
I20260808 22:51:49.982656 7759 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:51:49.982688 7759 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:51:49.982764 7759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:49.983511 7759 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42475 } }
I20260808 22:51:49.983678 7759 leader_election.cc:302] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3b2061b278b4a98b5d3bf841ee91ad6; no voters:
I20260808 22:51:49.983932 7759 leader_election.cc:288] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:51:49.984038 7769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:51:49.984297 7769 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [term 1 LEADER]: Becoming Leader. State: Replica: e3b2061b278b4a98b5d3bf841ee91ad6, State: Running, Role: LEADER
I20260808 22:51:49.984381 7759 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:51:49.984448 7769 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42475 } }
I20260808 22:51:49.984933 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42475 } } }
I20260808 22:51:49.985090 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:49.985363 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b2061b278b4a98b5d3bf841ee91ad6. Latest consensus state: current_term: 1 leader_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b2061b278b4a98b5d3bf841ee91ad6" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42475 } } }
I20260808 22:51:49.986092 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6 [sys.catalog]: This master's current role is: LEADER
I20260808 22:51:49.986804 7759 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 22:51:49.987123 7780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:51:49.991815 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:49.973684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.22.30.62:42475)
0808 22:51:49.974007 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:49.974018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:49.974086 (+ 68us) server_negotiation.cc:407] Connection header received
0808 22:51:49.974899 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:49.974910 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:49.974977 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:49.975141 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:49.976188 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:49.977817 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:49.979018 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:49.979856 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:49.979981 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:49.982896 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:49.982899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:49.982900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:49.982911 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:49.985516 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:49.986085 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:49.986088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:49.986088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:49.986255 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:49.986737 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:49.986738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:49.986738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:49.987080 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:49.987258 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:49.987446 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 22:51:49.987560 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
I20260808 22:51:49.995242 7780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:51:50.004235 7780 catalog_manager.cc:1347] Generated new cluster ID: d15e72392b904495a535551c3367e83a
I20260808 22:51:50.004283 7780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:51:50.040712 7780 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:51:50.041492 7780 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:51:50.046864 7780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6: Generated new TSK 0
I20260808 22:51:50.047081 7780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:51:50.873452 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:33149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42475
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:50.993700 7832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:50.993961 7832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:50.994001 7832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:50.995262 7832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:50.995317 7832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:50.995354 7832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:51:50.996834 7832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.30.20:33149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:50.997045 7832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:50.997278 7832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:51.003890 7839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:51.004081 7838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:51.003891 7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:51.004371 7832 server_base.cc:1034] running on GCE node
I20260808 22:51:51.004622 7832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:51.004948 7832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:51.006163 7832 hybrid_clock.cc:648] HybridClock initialized: now 1786229511006143 us; error 34 us; skew 500 ppm
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:51.009701 7832 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:51:51.010116 7832 webserver.cc:460] Webserver started at http://127.22.30.1:46857/ using document root <none> and password file <none>
I20260808 22:51:51.010380 7832 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:51.010450 7832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:51.010594 7832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:51:51.028036 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "d3546ec213ed475390250a3f6b769907"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "6e6ad27c3a7d3f9efb9b42adff3170e7"
server_key_iv: "ff1d5e1e74d2a25a06541043e213c77e"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.028609 7832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "d3546ec213ed475390250a3f6b769907"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "6e6ad27c3a7d3f9efb9b42adff3170e7"
server_key_iv: "ff1d5e1e74d2a25a06541043e213c77e"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.031237 7832 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 22:51:51.042765 7850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.042956 7832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:51:51.043074 7832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "d3546ec213ed475390250a3f6b769907"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "6e6ad27c3a7d3f9efb9b42adff3170e7"
server_key_iv: "ff1d5e1e74d2a25a06541043e213c77e"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.043166 7832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:51.057221 7832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:51.059051 7832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:51.059206 7832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:51.059481 7832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:51.059846 7832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:51.059911 7832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.059973 7832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:51.060001 7832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.077168 7832 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:44395
I20260808 22:51:51.077292 8040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:44395 every 8 connection(s)
I20260808 22:51:51.077687 7832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:51:51.086889 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 7832
I20260808 22:51:51.087023 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:51:51.087239 22648 external_mini_cluster.cc:1428] Setting key 4440f856105715b4d1b16887d51b5acd
I20260808 22:51:51.088338 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.078493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:43835 (local address 127.22.30.62:42475)
0808 22:51:51.078656 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.078658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.078909 (+ 251us) server_negotiation.cc:407] Connection header received
0808 22:51:51.079760 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.079761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.079774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.079837 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:51.080508 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.080992 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.082063 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.082254 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.082333 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.084894 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:51.084896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:51.084896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:51.084903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:51.086730 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.087176 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.087178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.087178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.087242 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.087568 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.087569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.087569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.087734 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:51.087822 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.088157 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.088245 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260808 22:51:51.088538 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.078502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42475 (local address 127.22.30.1:43835)
0808 22:51:51.078785 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:51.078820 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:51.079654 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:51.079898 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:51.079901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:51.080149 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:51.080416 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.080421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.081084 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.081085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.081961 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.081963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.082344 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.082345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.082496 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.083017 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:51.083024 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:51.084786 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:51.086810 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:51.086818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:51.086820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:51.087114 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:51.087315 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:51.087316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:51.087316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:51.087495 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:51.087821 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:51.087829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:51.088006 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 22:51:51.088112 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 22:51:51.088252 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":182,"threads_started":1}
I20260808 22:51:51.089231 8042 heartbeater.cc:347] Connected to a master server at 127.22.30.62:42475
I20260808 22:51:51.089337 8042 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:51.089505 8042 heartbeater.cc:511] Master 127.22.30.62:42475 requested a full tablet report, sending...
I20260808 22:51:51.090016 7698 ts_manager.cc:194] Registered new tserver with Master: d3546ec213ed475390250a3f6b769907 (127.22.30.1:44395)
I20260808 22:51:51.091065 7698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:43835
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:51:51.115476 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:33149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42475
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:51.242004 8054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:51.242260 8054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:51.242295 8054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:51.244176 8054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:51.244299 8054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:51.244402 8054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:51:51.246481 8054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.30.20:33149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:51.246878 8054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:51.247159 8054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:51.255674 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:51.255980 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:51.257311 8054 server_base.cc:1034] running on GCE node
W20260808 22:51:51.257514 8072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:51.257850 8054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:51.258113 8054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:51.259262 8054 hybrid_clock.cc:648] HybridClock initialized: now 1786229511259250 us; error 61 us; skew 500 ppm
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:51.262310 8054 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:51:51.262799 8054 webserver.cc:460] Webserver started at http://127.22.30.2:46677/ using document root <none> and password file <none>
I20260808 22:51:51.263006 8054 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:51.263069 8054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:51.263203 8054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:51:51.274108 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "09c2fcf0c20342a992c1771af8cd8847"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "153c66dde4e673c0e28ef67ed01f53d7"
server_key_iv: "0f7318a622326d7e44199afe283f9a29"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.274642 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "09c2fcf0c20342a992c1771af8cd8847"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "153c66dde4e673c0e28ef67ed01f53d7"
server_key_iv: "0f7318a622326d7e44199afe283f9a29"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.280308 8054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260808 22:51:51.303553 8084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.303803 8054 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260808 22:51:51.303896 8054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "09c2fcf0c20342a992c1771af8cd8847"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "153c66dde4e673c0e28ef67ed01f53d7"
server_key_iv: "0f7318a622326d7e44199afe283f9a29"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.303982 8054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:51.352543 8054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:51.355283 8054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:51.355548 8054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:51.356029 8054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:51.356537 8054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:51.356658 8054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.356732 8054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:51.356786 8054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.378209 8054 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:36773
I20260808 22:51:51.378854 8054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:51:51.379580 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 8054
I20260808 22:51:51.379742 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:51:51.379976 22648 external_mini_cluster.cc:1428] Setting key 3f164cf7cecc59eac8a4dc54fa3579fd
I20260808 22:51:51.381268 8265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:36773 every 8 connection(s)
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
I20260808 22:51:51.396010 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.382383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:44365 (local address 127.22.30.62:42475)
0808 22:51:51.383964 (+ 1581us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.383966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.383978 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:51:51.384284 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.384285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.384298 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.384362 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:51.385332 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.386158 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.388066 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.388417 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.388482 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.391688 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:51.391689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:51.391690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:51.391697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:51.393725 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.394431 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.394432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.394433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.394523 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.394894 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.394895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.394895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.395030 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:51.395252 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.395846 (+ 594us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.395923 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1552}
I20260808 22:51:51.396162 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.381956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42475 (local address 127.22.30.2:44365)
0808 22:51:51.382749 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:51.382797 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:51.384164 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:51.384431 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:51.384435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:51.384745 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:51.385213 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.385221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.386323 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.386324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.387935 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.387939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.388558 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.388559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.388788 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.389605 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:51.389617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:51.391581 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:51.393875 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:51.393881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:51.393883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:51.394301 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:51.394599 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:51.394600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:51.394601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:51.394817 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:51.395297 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:51.395304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:51.395601 (+ 297us) client_negotiation.cc:769] Sending connection context
0808 22:51:51.395717 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 22:51:51.395874 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":206,"threads_started":1}
I20260808 22:51:51.396853 8270 heartbeater.cc:347] Connected to a master server at 127.22.30.62:42475
I20260808 22:51:51.396970 8270 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:51.397202 8270 heartbeater.cc:511] Master 127.22.30.62:42475 requested a full tablet report, sending...
I20260808 22:51:51.397635 7698 ts_manager.cc:194] Registered new tserver with Master: 09c2fcf0c20342a992c1771af8cd8847 (127.22.30.2:36773)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
I20260808 22:51:51.398109 7698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:44365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:51:51.414345 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:33149/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42475
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:51:51.544217 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:51:51.544447 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:51:51.544544 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:51:51.546196 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:51:51.546296 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:51:51.546360 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:51:51.548424 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.30.20:33149/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:51:51.548696 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:51:51.548925 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:51:51.556658 8298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:51.557075 8299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:51:51.557348 8302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:51:51.558595 8285 server_base.cc:1034] running on GCE node
I20260808 22:51:51.558882 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:51:51.559166 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:51:51.560308 8285 hybrid_clock.cc:648] HybridClock initialized: now 1786229511560287 us; error 43 us; skew 500 ppm
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:51:51.563627 8285 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:51:51.564249 8285 webserver.cc:460] Webserver started at http://127.22.30.3:37035/ using document root <none> and password file <none>
I20260808 22:51:51.564620 8285 fs_manager.cc:359] Metadata directory not provided
I20260808 22:51:51.564703 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:51:51.564850 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:51:51.591120 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "1c8a33fc091d47c59ac695c56a614fb6"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "255fd62020e0018cd573d1a153bf2082"
server_key_iv: "1e202480c430b46034bcc7cc03c1a08e"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.591926 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "1c8a33fc091d47c59ac695c56a614fb6"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "255fd62020e0018cd573d1a153bf2082"
server_key_iv: "1e202480c430b46034bcc7cc03c1a08e"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.595194 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260808 22:51:51.631222 8311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.631503 8285 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260808 22:51:51.631664 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "1c8a33fc091d47c59ac695c56a614fb6"
format_stamp: "Formatted at 2026-08-08 22:51:51 on dist-test-slave-x4qh"
server_key: "255fd62020e0018cd573d1a153bf2082"
server_key_iv: "1e202480c430b46034bcc7cc03c1a08e"
server_key_version: "kuduclusterkey@0"
I20260808 22:51:51.631798 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:51:51.646015 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:51:51.648883 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:51:51.649083 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:51:51.649521 8285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:51:51.650072 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:51:51.650146 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.650180 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:51:51.650274 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:51:51.674182 8285 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:36161
I20260808 22:51:51.674862 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:51:51.675007 8502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:36161 every 8 connection(s)
I20260808 22:51:51.683848 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 8285
I20260808 22:51:51.683961 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:51:51.684196 22648 external_mini_cluster.cc:1428] Setting key 0f75fc0a0aca2ba6ff59fb8b79950aa8
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:51:51.688599 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.676146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:43785 (local address 127.22.30.62:42475)
0808 22:51:51.676187 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.676189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.677446 (+ 1257us) server_negotiation.cc:407] Connection header received
0808 22:51:51.678074 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.678076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.678088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.678116 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:51.679790 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.680545 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.682029 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.682350 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.682482 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.685551 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:51.685552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:51.685553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:51.685559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:51.686979 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.687549 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.687550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.687551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.687610 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.687858 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.687858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.687859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.687986 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:51.688060 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.688439 (+ 379us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.688518 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:51:51.688864 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.675891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42475 (local address 127.22.30.3:43785)
0808 22:51:51.677313 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:51.677359 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:51.677961 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:51.679196 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:51.679201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:51.679498 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:51:51.679691 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.679696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.680647 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.680648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.681922 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.681926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.682491 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.682492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.682654 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.683304 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:51:51.683314 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:51:51.685257 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:51:51.687065 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:51.687071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:51.687073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:51.687350 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:51.687675 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:51:51.687676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:51:51.687676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:51:51.687786 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:51:51.688052 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:51:51.688058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:51:51.688302 (+ 244us) client_negotiation.cc:769] Sending connection context
0808 22:51:51.688403 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:51:51.688535 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":77,"threads_started":1}
I20260808 22:51:51.689260 8504 heartbeater.cc:347] Connected to a master server at 127.22.30.62:42475
I20260808 22:51:51.689347 8504 heartbeater.cc:464] Registering TS with master...
I20260808 22:51:51.689502 8504 heartbeater.cc:511] Master 127.22.30.62:42475 requested a full tablet report, sending...
I20260808 22:51:51.689848 7698 ts_manager.cc:194] Registered new tserver with Master: 1c8a33fc091d47c59ac695c56a614fb6 (127.22.30.3:36161)
I20260808 22:51:51.690186 7698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:43785
I20260808 22:51:51.696604 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:51:51.705231 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.698717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.22.30.62:42475)
0808 22:51:51.698776 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.698778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.698962 (+ 184us) server_negotiation.cc:407] Connection header received
0808 22:51:51.699046 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.699047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.699054 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.699096 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:51:51.700112 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.700599 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.701319 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.701489 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.701559 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.702626 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:51:51.702627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:51:51.702628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:51:51.702633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:51:51.704003 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.704455 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.704456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.704457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.704533 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:51:51.704731 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:51:51.704732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:51:51.704732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:51:51.704839 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:51:51.704943 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.705062 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.705134 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:51:51.707350 7698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:51:51.709707 7698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:51:51.718752 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.712366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:36773 (local address 127.0.0.1:59702)
0808 22:51:51.713731 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:51.713755 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:51.713861 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:51.714295 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:51.714295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:51.714297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:51.714373 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.714375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.715724 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.715725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.717327 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.717330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.718468 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.718468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.718548 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.718560 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:51:51.718638 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:51:51.718686 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":185,"thread_start_us":1234,"threads_started":1}
I20260808 22:51:51.718762 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.713016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58380 (local address 127.22.30.1:44395)
0808 22:51:51.713187 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.713191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.713824 (+ 633us) server_negotiation.cc:407] Connection header received
0808 22:51:51.714000 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.714003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.714031 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.714103 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:51.714411 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.715625 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.717432 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.718339 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.718378 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.718423 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.718661 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.718700 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":124,"threads_started":1}
I20260808 22:51:51.718870 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.713237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.22.30.2:36773)
0808 22:51:51.713554 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.713559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.713821 (+ 262us) server_negotiation.cc:407] Connection header received
0808 22:51:51.714037 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.714039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.714056 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.714124 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:51.714456 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.715624 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.717455 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.718256 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.718330 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.718404 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.718689 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.718734 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":168,"threads_started":1}
I20260808 22:51:51.718942 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.712332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:44395 (local address 127.0.0.1:58380)
0808 22:51:51.713705 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:51.713755 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:51.713861 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:51.714239 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:51.714242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:51.714244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:51.714303 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.714305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.715722 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.715723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.717330 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.717332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.718471 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.718472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.718548 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.718560 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:51:51.718827 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 22:51:51.718872 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"mutex_wait_us":174,"thread_start_us":157,"threads_started":1}
I20260808 22:51:51.719893 8160 tablet_service.cc:1467] Processing CreateTablet for tablet be3c57a580b1484f88e820a0010325b0 (DEFAULT_TABLE table=test-table [id=40087926be624883bc2508d6422e0728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:51:51.720229 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3c57a580b1484f88e820a0010325b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:51.723487 8533 tablet_bootstrap.cc:492] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Bootstrap starting.
I20260808 22:51:51.724273 7933 tablet_service.cc:1467] Processing CreateTablet for tablet be3c57a580b1484f88e820a0010325b0 (DEFAULT_TABLE table=test-table [id=40087926be624883bc2508d6422e0728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:51:51.724589 8533 tablet_bootstrap.cc:654] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:51.724589 7933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3c57a580b1484f88e820a0010325b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:51.725277 8533 log.cc:824] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:51.725569 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.712372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36161 (local address 127.0.0.1:34522)
0808 22:51:51.716079 (+ 3707us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:51.716090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:51.716154 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:51.719167 (+ 3013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:51.719168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:51.719170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:51.719206 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.719208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.720480 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.720481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.721602 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:51.721605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.725231 (+ 3626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.725232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:51.725318 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.725324 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:51:51.725399 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:51:51.725449 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3690,"thread_start_us":239,"threads_started":1}
I20260808 22:51:51.725644 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:51.712717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.22.30.3:36161)
0808 22:51:51.713009 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:51.713013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:51.716148 (+ 3135us) server_negotiation.cc:407] Connection header received
0808 22:51:51.716268 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:51.716270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:51.716284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:51.716356 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:51.719267 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.720398 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.721774 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:51.722480 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:51.722556 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:51.722640 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:51.725434 (+ 2794us) server_negotiation.cc:299] Negotiation successful
0808 22:51:51.725507 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":182,"threads_started":1}
I20260808 22:51:51.728061 8394 tablet_service.cc:1467] Processing CreateTablet for tablet be3c57a580b1484f88e820a0010325b0 (DEFAULT_TABLE table=test-table [id=40087926be624883bc2508d6422e0728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:51:51.728372 8394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3c57a580b1484f88e820a0010325b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:51:51.728705 8535 tablet_bootstrap.cc:492] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: Bootstrap starting.
I20260808 22:51:51.731897 8535 tablet_bootstrap.cc:654] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:51.732722 8535 log.cc:824] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:51.734889 8533 tablet_bootstrap.cc:492] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: No bootstrap required, opened a new log
I20260808 22:51:51.734973 8533 ts_tablet_manager.cc:1397] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260808 22:51:51.737896 8535 tablet_bootstrap.cc:492] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: No bootstrap required, opened a new log
I20260808 22:51:51.737982 8535 ts_tablet_manager.cc:1397] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260808 22:51:51.739167 8533 raft_consensus.cc:348] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:51.739326 8533 raft_consensus.cc:374] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:51.739373 8533 raft_consensus.cc:729] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c2fcf0c20342a992c1771af8cd8847, State: Initialized, Role: FOLLOWER
I20260808 22:51:51.739499 8533 consensus_queue.cc:260] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:51.739856 8533 ts_tablet_manager.cc:1428] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260808 22:51:51.740022 8270 heartbeater.cc:503] Master 127.22.30.62:42475 was elected leader, sending a full tablet report...
I20260808 22:51:51.741652 8535 raft_consensus.cc:348] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:51.741791 8535 raft_consensus.cc:374] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:51.741822 8535 raft_consensus.cc:729] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3546ec213ed475390250a3f6b769907, State: Initialized, Role: FOLLOWER
I20260808 22:51:51.742317 8535 consensus_queue.cc:260] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:51.742918 8535 ts_tablet_manager.cc:1428] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260808 22:51:51.742951 8042 heartbeater.cc:503] Master 127.22.30.62:42475 was elected leader, sending a full tablet report...
I20260808 22:51:51.743386 8539 tablet_bootstrap.cc:492] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Bootstrap starting.
I20260808 22:51:51.744889 8539 tablet_bootstrap.cc:654] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Neither blocks nor log segments found. Creating new log.
I20260808 22:51:51.745210 8539 log.cc:824] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:51:51.746222 8539 tablet_bootstrap.cc:492] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: No bootstrap required, opened a new log
I20260808 22:51:51.746317 8539 ts_tablet_manager.cc:1397] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:51:51.750414 8539 raft_consensus.cc:348] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:51.750592 8539 raft_consensus.cc:374] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:51:51.750645 8539 raft_consensus.cc:729] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8a33fc091d47c59ac695c56a614fb6, State: Initialized, Role: FOLLOWER
I20260808 22:51:51.750802 8539 consensus_queue.cc:260] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:51.751201 8539 ts_tablet_manager.cc:1428] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260808 22:51:51.751407 8504 heartbeater.cc:503] Master 127.22.30.62:42475 was elected leader, sending a full tablet report...
W20260808 22:51:51.828928 8043 tablet.cc:2369] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:51:51.883081 8271 tablet.cc:2369] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:51:51.927498 8507 tablet.cc:2369] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 22:51:51 dist-test-slave-x4qh krb5kdc[6945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229511, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:51:52.100055 8541 raft_consensus.cc:482] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:51:52.100281 8541 raft_consensus.cc:504] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:52.100823 8541 leader_election.cc:288] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3546ec213ed475390250a3f6b769907 (127.22.30.1:44395), 1c8a33fc091d47c59ac695c56a614fb6 (127.22.30.3:36161)
I20260808 22:51:52.105382 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:52.101276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:36973 (local address 127.22.30.3:36161)
0808 22:51:52.101333 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:52.101336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:52.101528 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:51:52.101639 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:52.101641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:52.101650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:52.101677 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:52.101906 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.102648 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.104467 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.104976 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.105027 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:52.105037 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:52.105265 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 22:51:52.105310 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 22:51:52.105865 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:52.101086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:36161 (local address 127.22.30.2:36973)
0808 22:51:52.101466 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:52.101475 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:52.101554 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:52.101741 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:52.101742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:52.101744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:52.101833 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:52.101835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.102851 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.102852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:52.104340 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:52.104343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.105115 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.105116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:52.105174 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:52.105179 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:51:52.105692 (+ 513us) client_negotiation.cc:240] Negotiation successful
0808 22:51:52.105753 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":91,"threads_started":1}
I20260808 22:51:52.105937 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:52.101280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:47177 (local address 127.22.30.1:44395)
0808 22:51:52.101367 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:52.101369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:52.101571 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:51:52.101678 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:52.101679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:52.101694 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:52.101728 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:51:52.102056 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.102836 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.104601 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.105424 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.105497 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:52.105506 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:52.105803 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 22:51:52.105849 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 22:51:52.106292 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:52.100986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:44395 (local address 127.22.30.2:47177)
0808 22:51:52.101379 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:52.101392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:51:52.101603 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:51:52.101815 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:51:52.101816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:51:52.101818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:51:52.101942 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:52.101944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.102960 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.102961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:52.104456 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:51:52.104460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.105607 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.105608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:51:52.105714 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:52.105717 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:51:52.106065 (+ 348us) client_negotiation.cc:240] Negotiation successful
0808 22:51:52.106166 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":21,"thread_start_us":188,"threads_started":1}
I20260808 22:51:52.106696 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c57a580b1484f88e820a0010325b0" candidate_uuid: "09c2fcf0c20342a992c1771af8cd8847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8a33fc091d47c59ac695c56a614fb6" is_pre_election: true
I20260808 22:51:52.106714 7959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c57a580b1484f88e820a0010325b0" candidate_uuid: "09c2fcf0c20342a992c1771af8cd8847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3546ec213ed475390250a3f6b769907" is_pre_election: true
I20260808 22:51:52.106889 7959 raft_consensus.cc:2454] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c2fcf0c20342a992c1771af8cd8847 in term 0.
I20260808 22:51:52.106878 8425 raft_consensus.cc:2454] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c2fcf0c20342a992c1771af8cd8847 in term 0.
I20260808 22:51:52.107185 8088 leader_election.cc:302] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c2fcf0c20342a992c1771af8cd8847, 1c8a33fc091d47c59ac695c56a614fb6; no voters:
I20260808 22:51:52.107364 8541 raft_consensus.cc:2781] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:51:52.107439 8541 raft_consensus.cc:482] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:51:52.107461 8541 raft_consensus.cc:3037] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:52.108424 8541 raft_consensus.cc:504] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:52.108557 8541 leader_election.cc:288] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [CANDIDATE]: Term 1 election: Requested vote from peers d3546ec213ed475390250a3f6b769907 (127.22.30.1:44395), 1c8a33fc091d47c59ac695c56a614fb6 (127.22.30.3:36161)
I20260808 22:51:52.108788 7959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c57a580b1484f88e820a0010325b0" candidate_uuid: "09c2fcf0c20342a992c1771af8cd8847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3546ec213ed475390250a3f6b769907"
I20260808 22:51:52.108814 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c57a580b1484f88e820a0010325b0" candidate_uuid: "09c2fcf0c20342a992c1771af8cd8847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c8a33fc091d47c59ac695c56a614fb6"
I20260808 22:51:52.108920 7959 raft_consensus.cc:3037] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:52.108980 8425 raft_consensus.cc:3037] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:51:52.109978 7959 raft_consensus.cc:2454] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c2fcf0c20342a992c1771af8cd8847 in term 1.
I20260808 22:51:52.110263 8085 leader_election.cc:302] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c2fcf0c20342a992c1771af8cd8847, d3546ec213ed475390250a3f6b769907; no voters:
I20260808 22:51:52.110458 8541 raft_consensus.cc:2781] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:51:52.110606 8425 raft_consensus.cc:2454] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c2fcf0c20342a992c1771af8cd8847 in term 1.
I20260808 22:51:52.110632 8541 raft_consensus.cc:686] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 1 LEADER]: Becoming Leader. State: Replica: 09c2fcf0c20342a992c1771af8cd8847, State: Running, Role: LEADER
I20260808 22:51:52.110726 8541 consensus_queue.cc:237] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } }
I20260808 22:51:52.111789 7698 catalog_manager.cc:5495] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c2fcf0c20342a992c1771af8cd8847 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "09c2fcf0c20342a992c1771af8cd8847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c2fcf0c20342a992c1771af8cd8847" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 36773 } health_report { overall_health: HEALTHY } } }
I20260808 22:51:52.159498 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:52.156914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.22.30.2:36773)
0808 22:51:52.156974 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:51:52.156976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:51:52.157024 (+ 48us) server_negotiation.cc:407] Connection header received
0808 22:51:52.157070 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:51:52.157071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:51:52.157098 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:51:52.157244 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:51:52.157402 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.157857 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.158614 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:51:52.158817 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:51:52.158891 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:51:52.159040 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:51:52.159143 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:51:52.159224 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 22:51:52.159265 (+ 41us) server_negotiation.cc:299] Negotiation successful
0808 22:51:52.159292 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:51:52.161435 8425 raft_consensus.cc:1261] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 1 FOLLOWER]: Refusing update from remote peer 09c2fcf0c20342a992c1771af8cd8847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:51:52.161435 7959 raft_consensus.cc:1261] T be3c57a580b1484f88e820a0010325b0 P d3546ec213ed475390250a3f6b769907 [term 1 FOLLOWER]: Refusing update from remote peer 09c2fcf0c20342a992c1771af8cd8847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:51:52.161746 8541 consensus_queue.cc:1035] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3546ec213ed475390250a3f6b769907" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:51:52.161906 8551 consensus_queue.cc:1035] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c8a33fc091d47c59ac695c56a614fb6" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:51:52.164932 8556 mvcc.cc:205] Tried to move back new op lower bound from 7316396081810132992 to 7316396081606291456. Current Snapshot: MvccSnapshot[applied={T|T < 7316396081810132992}]
I20260808 22:51:52.165794 8558 mvcc.cc:205] Tried to move back new op lower bound from 7316396081810132992 to 7316396081606291456. Current Snapshot: MvccSnapshot[applied={T|T < 7316396081810132992}]
I20260808 22:51:52.168146 7697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:51:52.168236 7697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:51:52.168511 8557 mvcc.cc:205] Tried to move back new op lower bound from 7316396081810132992 to 7316396081606291456. Current Snapshot: MvccSnapshot[applied={T|T < 7316396081810132992}]
I20260808 22:51:52.169013 7697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3b2061b278b4a98b5d3bf841ee91ad6: Sending DeleteTablet for 3 replicas of tablet be3c57a580b1484f88e820a0010325b0
I20260808 22:51:52.169339 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet be3c57a580b1484f88e820a0010325b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:51:52 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:59702
I20260808 22:51:52.169409 7933 tablet_service.cc:1514] Processing DeleteTablet for tablet be3c57a580b1484f88e820a0010325b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:51:52 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:58380
I20260808 22:51:52.169682 8394 tablet_service.cc:1514] Processing DeleteTablet for tablet be3c57a580b1484f88e820a0010325b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:51:52 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:34522
I20260808 22:51:52.169988 8568 tablet_replica.cc:330] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: stopping tablet replica
I20260808 22:51:52.170149 8568 raft_consensus.cc:2229] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:51:52.170168 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 7832
I20260808 22:51:52.170181 8570 tablet_replica.cc:330] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: stopping tablet replica
I20260808 22:51:52.173069 8570 raft_consensus.cc:2229] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:51:52.173300 8570 raft_consensus.cc:2258] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:51:52.174489 8570 ts_tablet_manager.cc:1905] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:51:52.174839 8568 raft_consensus.cc:2258] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:51:52.175490 8568 ts_tablet_manager.cc:1905] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:51:52.178159 8568 ts_tablet_manager.cc:1918] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:51:52.178306 8568 log.cc:1196] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/be3c57a580b1484f88e820a0010325b0
I20260808 22:51:52.178676 8568 ts_tablet_manager.cc:1939] T be3c57a580b1484f88e820a0010325b0 P 09c2fcf0c20342a992c1771af8cd8847: Deleting consensus metadata
I20260808 22:51:52.179178 7678 catalog_manager.cc:4841] TS 09c2fcf0c20342a992c1771af8cd8847 (127.22.30.2:36773): tablet be3c57a580b1484f88e820a0010325b0 (table test-table [id=40087926be624883bc2508d6422e0728]) successfully deleted
W20260808 22:51:52.183198 7677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:44395 (error 108)
I20260808 22:51:52.183880 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 8054
I20260808 22:51:52.189873 8570 ts_tablet_manager.cc:1918] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:51:52.190080 8570 log.cc:1196] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/be3c57a580b1484f88e820a0010325b0
I20260808 22:51:52.190534 8570 ts_tablet_manager.cc:1939] T be3c57a580b1484f88e820a0010325b0 P 1c8a33fc091d47c59ac695c56a614fb6: Deleting consensus metadata
I20260808 22:51:52.191550 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:51:52.191413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:44395 (local address 127.0.0.1:58396)
0808 22:51:52.191467 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 22:51:52.191516 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:44395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 22:51:52.191742 7677 catalog_manager.cc:4584] TS d3546ec213ed475390250a3f6b769907 (127.22.30.1:44395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be3c57a580b1484f88e820a0010325b0: Network error: Client connection negotiation failed: client connection to 127.22.30.1:44395: connect: Connection refused (error 111)
I20260808 22:51:52.191864 7679 catalog_manager.cc:4841] TS 1c8a33fc091d47c59ac695c56a614fb6 (127.22.30.3:36161): tablet be3c57a580b1484f88e820a0010325b0 (table test-table [id=40087926be624883bc2508d6422e0728]) successfully deleted
I20260808 22:51:52.196664 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 8285
I20260808 22:51:52.203876 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 7655
I20260808 22:51:52.215862 22648 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 22:51:52.561017 22648 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 22:51:52.561141 22648 mini_ranger.cc:66] Stopping Ranger...
I20260808 22:51:52.906414 22648 mini_ranger.cc:68] Stopped Ranger
2026-08-08 22:51:52.906 UTC [6998] LOG: received smart shutdown request
2026-08-08 22:51:52.910 UTC [6998] LOG: background worker "logical replication launcher" (PID 7015) exited with exit code 1
2026-08-08 22:51:52.910 UTC [7004] LOG: shutting down
2026-08-08 22:51:53.038 UTC [6998] LOG: database system is shut down
2026-08-08T22:51:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44374 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:51:53 dist-test-slave-x4qh krb5kdc[8585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:51:53 dist-test-slave-x4qh krb5kdc[8585](info): set up 2 sockets
Aug 08 22:51:53 dist-test-slave-x4qh krb5kdc[8585](info): commencing operation
krb5kdc: starting...
W20260808 22:51:54.328336 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:51:54 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:51:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/rangeradmin_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/rangeradmin_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/rangerlookup_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/rangerlookup_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
I20260808 22:51:54.476028 22648 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskT1Dm48/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/postgres -l logfile start
2026-08-08 22:51:56.624 UTC [8637] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 22:51:56.624 UTC [8637] LOG: listening on IPv4 address "127.22.30.20", port 48119
2026-08-08 22:51:56.627 UTC [8637] LOG: listening on Unix socket "/tmp/.s.PGSQL.48119"
2026-08-08 22:51:56.646 UTC [8644] LOG: database system was shut down at 2026-08-08 22:51:55 UTC
2026-08-08 22:51:56.649 UTC [8637] LOG: database system is ready to accept connections
I20260808 22:51:57.714959 22648 mini_postgres.cc:96] Postgres bound to 48119
2026-08-08 22:51:57.721 UTC [8654] FATAL: database "slave" does not exist
127.22.30.20:48119 - accepting connections
I20260808 22:51:57.722313 22648 mini_ranger.cc:161] Starting Ranger...
I20260808 22:51:57.735948 22648 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 22:51:58.128312 22648 mini_ranger.cc:87] Created ranger Postgres database
I20260808 22:51:58.128401 22648 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-admin
2026-08-08 22:51:58,400 [I] DB FLAVOR :POSTGRES
2026-08-08 22:51:58,400 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:51:58,400 [I] Checking connection..
2026-08-08 22:51:58,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 22:51:58,644 [I] Checking connection passed.
2026-08-08 22:51:58,644 [I] --------- Verifying version history table ---------
2026-08-08 22:51:58,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:51:58,868 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:51:58,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:51:59,098 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 22:51:59,098 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 22:51:59,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 22:51:59.344 UTC [8745] WARNING: there is no transaction in progress
2026-08-08 22:51:59,352 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 22:51:59,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 22:51:59,598 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 22:51:59,598 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:51:59,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 22:51:59,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 22:52:00,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-x4qh.c.gcp-upstream.internal','N') ;"
2026-08-08 22:52:00,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:52:00,530 [I] Table x_portal_user does not exist in database ranger
2026-08-08 22:52:00,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:52:00,774 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 22:52:00,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:52:01,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:52:01,244 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 22:52:01,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 22:52:02.128 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.143 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.157 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.171 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.185 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.200 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.250 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.255 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.268 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.277 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.286 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.299 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.308 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.313 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.319 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02.325 UTC [8910] WARNING: there is no transaction in progress
2026-08-08 22:52:02,863 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 22:52:02,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 22:52:03,102 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 22:52:03,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 22:52:03,319 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 22:52:03,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 22:52:03,570 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 22:52:03,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 22:52:03,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 22:52:04,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-x4qh.c.gcp-upstream.internal';"
2026-08-08 22:52:04,279 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 22:52:04,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 22:52:04,523 [I] DB_PATCHES have already been applied
I20260808 22:52:04.528559 22648 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-admin:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskT1Dm48/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 22:52:04.528636 22648 mini_ranger.cc:193] Using host: 127.22.30.20
I20260808 22:52:04.532287 22648 mini_ranger.cc:227] Ranger admin URL: http://127.22.30.20:46579
Aug 08, 2026 10:52:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:52:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:52:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 10:52:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 10:52:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 10:52:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 10:52:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 22:52:05.799026 22648 mini_ranger.cc:160] Time spent starting Ranger: real 8.077s user 0.001s sys 0.008s
Aug 08, 2026 10:52:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 10:52:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 10:52:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 10:52:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 10:52:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 22:52:26.027449 22648 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.30.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.30.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/rangerkms_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.30.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/rangerkms_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.30.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.30.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.30.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/HTTP_127.22.30.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 22:52:26 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229546, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 22:52:26.098505 22648 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 22:52:26.107085 22648 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 22:52:26.836628 22648 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 22:52:26.836719 22648 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms
I20260808 22:52:26.836741 22648 mini_ranger_kms.cc:227] Using postgres at 127.22.30.20:48119
2026-08-08 22:52:27,000 [I] DB FLAVOR :POSTGRES
2026-08-08 22:52:27,000 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 22:52:27,000 [I] Checking connection
2026-08-08 22:52:27,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 22:52:27,221 [I] connection success
2026-08-08 22:52:27,221 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 22:52:27,221 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 22:52:27,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 22:52:27,466 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 22:52:27,467 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 22:52:27,467 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 22:52:27,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.30.20:48119/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 22:52:27,727 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 22:52:28.131263 22648 mini_ranger_kms.cc:310] Created kms service
I20260808 22:52:28.296032 22648 mini_ranger_kms.cc:326] Created kudu user
I20260808 22:52:28.335496 22648 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 22:52:28 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229546, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:28.713965 22648 mini_ranger_kms.cc:384] Added ranger policy
I20260808 22:52:28.714079 22648 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms:/tmp/dist-test-taskT1Dm48/build/release/bin/postgresql.jar:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskT1Dm48/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 22:52:28.714097 22648 mini_ranger_kms.cc:240] Using host: 127.22.30.20
I20260808 22:52:28.717975 22648 mini_ranger_kms.cc:276] Ranger KMS PID: 9181
I20260808 22:52:28.718051 22648 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.30.20:57607
Aug 08, 2026 10:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 10:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 10:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 10:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 10:52:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 10:52:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 22:52:30 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786229550, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.30.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:31.310602 22648 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.212s user 0.001s sys 0.147s
I20260808 22:52:31.310710 22648 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 22:52:31.310760 22648 mini_ranger_kms.cc:401] 127.22.30.20:57607/kms/v1/keys
Aug 08 22:52:31 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229550, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.30.20@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
Aug 08 22:52:31 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229550, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.30.20@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
Aug 08 22:52:33 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:52:33.752034 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:45377
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:45377
--ranger_config_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:33.827333 9259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:33.827586 9259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:33.827608 9259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:33.828691 9259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:33.828730 9259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:33.828744 9259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:33.828756 9259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:33.828768 9259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:33.830233 9259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.30.20:57607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:45377
--ranger_config_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:33.830443 9259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:33.830751 9259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:33.835466 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:33.835639 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:33.835664 9259 server_base.cc:1034] running on GCE node
W20260808 22:52:33.835486 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:33.835994 9259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:33.836297 9259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:33.837457 9259 hybrid_clock.cc:648] HybridClock initialized: now 1786229553837444 us; error 29 us; skew 500 ppm
Aug 08 22:52:33 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229553, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:33.845168 9259 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:33.845662 9259 webserver.cc:460] Webserver started at http://127.22.30.62:38191/ using document root <none> and password file <none>
I20260808 22:52:33.845908 9259 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:33.845947 9259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:33.846073 9259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:52:33 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229553, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:33.888572 9259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "2a9673005d2743e982c7783c685f08fb"
format_stamp: "Formatted at 2026-08-08 22:52:33 on dist-test-slave-x4qh"
server_key: "f9f152567afd4d62b2f99548e89eb5b4"
server_key_iv: "06999bcbe5a821cc24ba84d07976fdae"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:33.888972 9259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "2a9673005d2743e982c7783c685f08fb"
format_stamp: "Formatted at 2026-08-08 22:52:33 on dist-test-slave-x4qh"
server_key: "f9f152567afd4d62b2f99548e89eb5b4"
server_key_iv: "06999bcbe5a821cc24ba84d07976fdae"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:33.890365 9259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 22:52:33.900565 9283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:33.900765 9259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:52:33.900882 9259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "2a9673005d2743e982c7783c685f08fb"
format_stamp: "Formatted at 2026-08-08 22:52:33 on dist-test-slave-x4qh"
server_key: "f9f152567afd4d62b2f99548e89eb5b4"
server_key_iv: "06999bcbe5a821cc24ba84d07976fdae"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:33.900955 9259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:33.913455 9259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:33.914824 9259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:33.914969 9259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:33.918846 9259 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:45377
I20260808 22:52:33.918926 9364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:45377 every 8 connection(s)
I20260808 22:52:33.919241 9259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:33.919874 9366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:33.921928 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Bootstrap starting.
I20260808 22:52:33.922461 9366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:33.922745 9366 log.cc:824] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:33.923362 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: No bootstrap required, opened a new log
I20260808 22:52:33.925241 9366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:33.925372 9366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:33.925422 9366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9673005d2743e982c7783c685f08fb, State: Initialized, Role: FOLLOWER
I20260808 22:52:33.925549 9366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:33.925611 9366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:52:33.925642 9366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:52:33.925681 9366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:33.926195 9366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:33.926301 9366 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9673005d2743e982c7783c685f08fb; no voters:
I20260808 22:52:33.926465 9366 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:52:33.926498 9371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:33.926648 9371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 LEADER]: Becoming Leader. State: Replica: 2a9673005d2743e982c7783c685f08fb, State: Running, Role: LEADER
I20260808 22:52:33.926707 9366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:33.926717 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 9259
I20260808 22:52:33.926810 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:52:33.926766 9371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:33.926996 22648 external_mini_cluster.cc:1428] Setting key d3db787c50d7674898d3bf62c2b49f9e
I20260808 22:52:33.927062 9376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9673005d2743e982c7783c685f08fb. Latest consensus state: current_term: 1 leader_uuid: "2a9673005d2743e982c7783c685f08fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } } }
I20260808 22:52:33.927155 9376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:33.927057 9372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a9673005d2743e982c7783c685f08fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } } }
I20260808 22:52:33.927230 9372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:33.927405 9382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:33.928020 9382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:33.928459 9366 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 22:52:33.932814 9382 catalog_manager.cc:1347] Generated new cluster ID: 8ed96d73b8474a2db0da2ecfbca537d0
I20260808 22:52:33.932895 9382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:52:33 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:33.940483 9382 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:52:33.941253 9382 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:52:33.942184 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:33.927605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.22.30.62:45377)
0808 22:52:33.927807 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:33.927817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:33.928102 (+ 285us) server_negotiation.cc:407] Connection header received
0808 22:52:33.928234 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:33.928242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:33.928311 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:33.928447 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:33.929283 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:33.933068 (+ 3785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:33.934126 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:33.934723 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:33.934833 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:33.937952 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:33.937957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:33.937957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:33.937969 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:33.940058 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:33.940505 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:33.940508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:33.940508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:33.940640 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:33.940879 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:33.940880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:33.940880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:33.941169 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:33.941296 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:33.941848 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 22:52:33.941993 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260808 22:52:33.953429 9382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Generated new TSK 0
I20260808 22:52:33.953584 9382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:52:34.593974 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:45377
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:34.699265 9449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:34.699507 9449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:34.699540 9449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:34.700842 9449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:34.700911 9449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:34.700953 9449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:52:34.702247 9449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.30.20:57607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:34.702467 9449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:34.702679 9449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:34.708431 9460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:34.708398 9456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:34.708487 9464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:34.708693 9449 server_base.cc:1034] running on GCE node
I20260808 22:52:34.709048 9449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:34.709264 9449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:34.710405 9449 hybrid_clock.cc:648] HybridClock initialized: now 1786229554710383 us; error 39 us; skew 500 ppm
Aug 08 22:52:34 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:34.713047 9449 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:52:34.713507 9449 webserver.cc:460] Webserver started at http://127.22.30.1:42683/ using document root <none> and password file <none>
I20260808 22:52:34.713750 9449 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:34.713806 9449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:34.713919 9449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:52:34 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:34.725730 9449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "8b023fa1d71c421f8fafdfca04c01ad4"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "5ddd736d0e1b1196f548be2f507e5550"
server_key_iv: "60a687c71d5824daebf279a03d73bc9d"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:34.726238 9449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "8b023fa1d71c421f8fafdfca04c01ad4"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "5ddd736d0e1b1196f548be2f507e5550"
server_key_iv: "60a687c71d5824daebf279a03d73bc9d"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:34.728317 9449 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:52:34.742830 9477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:34.743039 9449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 22:52:34.743122 9449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "8b023fa1d71c421f8fafdfca04c01ad4"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "5ddd736d0e1b1196f548be2f507e5550"
server_key_iv: "60a687c71d5824daebf279a03d73bc9d"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:34.743196 9449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:34.774529 9449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:34.776432 9449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:34.776562 9449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:34.776804 9449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:34.777130 9449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:34.777163 9449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:34.777181 9449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:34.777192 9449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:34.783893 9449 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:39755
I20260808 22:52:34.784296 9449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:52:34.785027 9659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:39755 every 8 connection(s)
I20260808 22:52:34.791759 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 9449
I20260808 22:52:34.791857 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:52:34.792034 22648 external_mini_cluster.cc:1428] Setting key 77f7594724313bbcdf6294057a547f7a
Aug 08 22:52:34 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:34.797166 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:34.785235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:45377 (local address 127.22.30.1:50115)
0808 22:52:34.785400 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:34.785442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:34.786173 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:34.787818 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:34.787823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:34.788066 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:34.788323 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:34.788329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:34.789343 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:34.789345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:34.790579 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:34.790582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:34.791101 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:34.791102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:34.791222 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:34.791659 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:34.791668 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:34.793370 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:34.795683 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:34.795690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:34.795692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:34.795955 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:34.796162 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:34.796163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:34.796164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:34.796291 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:34.796557 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:34.796562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:34.796740 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 22:52:34.796819 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 22:52:34.796948 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260808 22:52:34.797875 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:34.785550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:50115 (local address 127.22.30.62:45377)
0808 22:52:34.787482 (+ 1932us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:34.787484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:34.787494 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:52:34.787520 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:34.787521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:34.787533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:34.787732 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:34.788445 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:34.789238 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:34.790694 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:34.791006 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:34.791434 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:34.793469 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:34.793471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:34.793471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:34.793478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:34.795559 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:34.796039 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:34.796040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:34.796040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:34.796097 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:34.796361 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:34.796361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:34.796362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:34.796492 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:34.797644 (+ 1152us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:34.797686 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 22:52:34.797765 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1901,"thread_start_us":81,"threads_started":1}
I20260808 22:52:34.798441 9661 heartbeater.cc:347] Connected to a master server at 127.22.30.62:45377
I20260808 22:52:34.798545 9661 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:34.798743 9661 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:34.799292 9310 ts_manager.cc:194] Registered new tserver with Master: 8b023fa1d71c421f8fafdfca04c01ad4 (127.22.30.1:39755)
I20260808 22:52:34.800164 9310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:50115
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:52:34.830338 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:45377
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:52:34 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 22:52:34.951753 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:34.952005 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:34.952041 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:34.953604 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:34.953680 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:34.953758 9676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:52:34.955158 9676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.30.20:57607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:34.955477 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:34.955716 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:34.961494 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:34.961639 9676 server_base.cc:1034] running on GCE node
W20260808 22:52:34.961673 9683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:34.961848 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:34.962105 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:34.962352 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:34.964169 9676 hybrid_clock.cc:648] HybridClock initialized: now 1786229554964140 us; error 44 us; skew 500 ppm
Aug 08 22:52:34 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:34.967249 9676 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:52:34.967717 9676 webserver.cc:460] Webserver started at http://127.22.30.2:35735/ using document root <none> and password file <none>
I20260808 22:52:34.968009 9676 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:34.968079 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:34.968225 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:52:34 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:34.981025 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "ffa4d7f957744efabf65df8ae2329646"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "508ebf9452a49386d7cf1c75170a2d42"
server_key_iv: "82e77bdc1a143a3ba03734478627ed3a"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:34.981529 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "ffa4d7f957744efabf65df8ae2329646"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "508ebf9452a49386d7cf1c75170a2d42"
server_key_iv: "82e77bdc1a143a3ba03734478627ed3a"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:34.987766 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260808 22:52:35.008749 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.009094 9676 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260808 22:52:35.009327 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "ffa4d7f957744efabf65df8ae2329646"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "508ebf9452a49386d7cf1c75170a2d42"
server_key_iv: "82e77bdc1a143a3ba03734478627ed3a"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.009501 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:35.060550 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:35.063232 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:35.063484 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:35.063777 9676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:35.064210 9676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:35.064287 9676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.064406 9676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:35.064471 9676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.082908 9676 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:37273
I20260808 22:52:35.083648 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:52:35.086392 9867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:37273 every 8 connection(s)
I20260808 22:52:35.091720 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 9676
I20260808 22:52:35.091822 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:52:35.091993 22648 external_mini_cluster.cc:1428] Setting key 7aa495be788eb9acfde5365f3d200768
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:35.102630 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:35.086249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:43253 (local address 127.22.30.62:45377)
0808 22:52:35.086295 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:35.086297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:35.086978 (+ 681us) server_negotiation.cc:407] Connection header received
0808 22:52:35.088230 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:35.088232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:35.088262 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:35.088297 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:35.089630 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.090426 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.095314 (+ 4888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.095620 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.095690 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:35.099065 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:35.099066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:35.099067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:35.099074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:35.100888 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:35.101546 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:35.101547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:35.101547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:35.101602 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:35.101865 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:35.101866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:35.101866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:35.101984 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:35.102022 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:35.102429 (+ 407us) server_negotiation.cc:299] Negotiation successful
0808 22:52:35.102526 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:52:35.103411 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:35.085929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:45377 (local address 127.22.30.2:43253)
0808 22:52:35.086797 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:35.086841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:35.088063 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:35.088500 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:35.088505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:35.089017 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:35.089487 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:35.089495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.090544 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.090544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:35.092106 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:35.092109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.095840 (+ 3731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.095841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:35.096023 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:35.096796 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:35.096806 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:35.098951 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:35.101004 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:35.101010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:35.101012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:35.101378 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:35.101680 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:35.101680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:35.101680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:35.101796 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:35.102065 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:35.102070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:35.102282 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 22:52:35.102960 (+ 678us) client_negotiation.cc:240] Negotiation successful
0808 22:52:35.103118 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":126,"threads_started":1}
I20260808 22:52:35.104180 9868 heartbeater.cc:347] Connected to a master server at 127.22.30.62:45377
I20260808 22:52:35.104269 9868 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:35.104460 9868 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:35.104884 9310 ts_manager.cc:194] Registered new tserver with Master: ffa4d7f957744efabf65df8ae2329646 (127.22.30.2:37273)
I20260808 22:52:35.105340 9310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:43253
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:52:35.130213 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:45377
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:35.256714 9875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:35.257014 9875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:35.257086 9875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:35.258703 9875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:35.258797 9875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:35.258875 9875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:52:35.260841 9875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.30.20:57607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:35.261237 9875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:35.261497 9875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:35.267781 9884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:35.268172 9892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.269140 9875 server_base.cc:1034] running on GCE node
W20260808 22:52:35.269193 9883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.269461 9875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:35.269685 9875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:35.270834 9875 hybrid_clock.cc:648] HybridClock initialized: now 1786229555270808 us; error 38 us; skew 500 ppm
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:35.274935 9875 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:52:35.275537 9875 webserver.cc:460] Webserver started at http://127.22.30.3:37155/ using document root <none> and password file <none>
I20260808 22:52:35.275780 9875 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:35.275825 9875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:35.275931 9875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:35.297904 9875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "c8b480c38c9d423e92fbdccbab9defa0"
format_stamp: "Formatted at 2026-08-08 22:52:35 on dist-test-slave-x4qh"
server_key: "427ed8e952e5a0bd01a5dcb60ea96d75"
server_key_iv: "e8070b4de5e9881d7e85cdf5c4114fa6"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.298444 9875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "c8b480c38c9d423e92fbdccbab9defa0"
format_stamp: "Formatted at 2026-08-08 22:52:35 on dist-test-slave-x4qh"
server_key: "427ed8e952e5a0bd01a5dcb60ea96d75"
server_key_iv: "e8070b4de5e9881d7e85cdf5c4114fa6"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.305927 9875 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.001s
I20260808 22:52:35.327590 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.331514 9875 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.000s sys 0.001s
I20260808 22:52:35.331634 9875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "c8b480c38c9d423e92fbdccbab9defa0"
format_stamp: "Formatted at 2026-08-08 22:52:35 on dist-test-slave-x4qh"
server_key: "427ed8e952e5a0bd01a5dcb60ea96d75"
server_key_iv: "e8070b4de5e9881d7e85cdf5c4114fa6"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.331723 9875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:35.345839 9875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:35.349836 9875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:35.350021 9875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:35.350293 9875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:35.350816 9875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:35.350905 9875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.350971 9875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:35.351025 9875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.363469 9875 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:46599
I20260808 22:52:35.366443 9875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:52:35.368505 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 9875
I20260808 22:52:35.368613 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:52:35.368786 22648 external_mini_cluster.cc:1428] Setting key 6854f2c378cf8a972b8ff69c2483475f
I20260808 22:52:35.372148 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:46599 every 8 connection(s)
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:35.385918 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:35.372867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36001 (local address 127.22.30.62:45377)
0808 22:52:35.372913 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:35.372916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:35.375012 (+ 2096us) server_negotiation.cc:407] Connection header received
0808 22:52:35.375728 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:35.375729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:35.375740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:35.375768 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:35.376574 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.377316 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.378749 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.379062 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.379111 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:35.382806 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:35.382807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:35.382808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:35.382814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:35.384347 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:35.384854 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:35.384856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:35.384856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:35.384914 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:35.385176 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:35.385177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:35.385177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:35.385304 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:35.385345 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:35.385740 (+ 395us) server_negotiation.cc:299] Negotiation successful
0808 22:52:35.385820 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:52:35.389748 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:35.372876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:45377 (local address 127.22.30.3:36001)
0808 22:52:35.373119 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:35.374926 (+ 1807us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:35.375622 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:35.375949 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:35.375953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:35.376177 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:35.376470 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:35.376476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.377427 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.377428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:35.378641 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:35.378644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.379222 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.379223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:35.379359 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:35.380468 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:35.380477 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:35.382708 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:35.384458 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:35.384464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:35.384467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:35.384746 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:35.384980 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:35.384980 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:35.384981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:35.385102 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:35.385391 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:35.385396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:35.385577 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 22:52:35.386449 (+ 872us) client_negotiation.cc:240] Negotiation successful
0808 22:52:35.386602 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260808 22:52:35.390182 10088 heartbeater.cc:347] Connected to a master server at 127.22.30.62:45377
I20260808 22:52:35.390275 10088 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:35.390455 10088 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:35.390782 9310 ts_manager.cc:194] Registered new tserver with Master: c8b480c38c9d423e92fbdccbab9defa0 (127.22.30.3:46599)
I20260808 22:52:35.391250 9310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:36001
I20260808 22:52:35.391587 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:52:35.392311 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 9449
I20260808 22:52:35.420910 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 9676
I20260808 22:52:35.432859 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 9875
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229554, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:35.453761 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 9259
I20260808 22:52:35.467993 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:45377
--webserver_interface=127.22.30.62
--webserver_port=38191
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:45377
--ranger_config_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:35.551203 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:35.551430 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:35.551469 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:35.552510 10092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:35.552564 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:35.552588 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:35.552636 10092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:35.552663 10092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:35.554111 10092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:45377
--ranger_config_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=38191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:35.554332 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:35.554512 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:35.558892 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:35.558892 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:35.559154 10111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.559147 10092 server_base.cc:1034] running on GCE node
I20260808 22:52:35.559465 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:35.559698 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:35.560840 10092 hybrid_clock.cc:648] HybridClock initialized: now 1786229555560829 us; error 27 us; skew 500 ppm
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:35.563002 10092 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:35.563454 10092 webserver.cc:460] Webserver started at http://127.22.30.62:38191/ using document root <none> and password file <none>
I20260808 22:52:35.563688 10092 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:35.563737 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:35.605172 10092 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:52:35.605895 10122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.606062 10092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:52:35.606176 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "2a9673005d2743e982c7783c685f08fb"
format_stamp: "Formatted at 2026-08-08 22:52:33 on dist-test-slave-x4qh"
server_key: "f9f152567afd4d62b2f99548e89eb5b4"
server_key_iv: "06999bcbe5a821cc24ba84d07976fdae"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.606588 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:35.644657 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:35.646611 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:35.646767 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:35.651639 10092 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:45377
I20260808 22:52:35.651723 10204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:45377 every 8 connection(s)
I20260808 22:52:35.652050 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:35.653630 10205 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:52:35.654467 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Bootstrap starting.
I20260808 22:52:35.656230 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 10092
I20260808 22:52:35.656507 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:39755
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:45377
--webserver_port=42683
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:35.662933 10205 log.cc:824] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:35.664497 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:52:35.664772 10205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Bootstrap complete.
I20260808 22:52:35.666893 10205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:35.667090 10205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9673005d2743e982c7783c685f08fb, State: Initialized, Role: FOLLOWER
I20260808 22:52:35.667248 10205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:35.667301 10205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:52:35.667349 10205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:52:35.667411 10205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:52:35.668320 10205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:35.668426 10205 leader_election.cc:302] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a9673005d2743e982c7783c685f08fb; no voters:
I20260808 22:52:35.668610 10205 leader_election.cc:288] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:52:35.668651 10211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:52:35.668769 10211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [term 2 LEADER]: Becoming Leader. State: Replica: 2a9673005d2743e982c7783c685f08fb, State: Running, Role: LEADER
I20260808 22:52:35.668859 10205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:35.668860 10211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } }
I20260808 22:52:35.669287 10212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9673005d2743e982c7783c685f08fb. Latest consensus state: current_term: 2 leader_uuid: "2a9673005d2743e982c7783c685f08fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } } }
I20260808 22:52:35.669486 10212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:35.669567 10213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a9673005d2743e982c7783c685f08fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a9673005d2743e982c7783c685f08fb" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 45377 } } }
I20260808 22:52:35.669642 10213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:35.669701 10224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:35.670145 10224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:35.670264 10224 catalog_manager.cc:1259] Loaded cluster ID: 8ed96d73b8474a2db0da2ecfbca537d0
I20260808 22:52:35.670305 10224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:52:35.670778 10205 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 22:52:35.675228 10224 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:52:35.675769 10224 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Loaded TSK: 0
I20260808 22:52:35.676016 10224 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:52:35.761389 10208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:35.761622 10208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:35.761654 10208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:35.763160 10208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:35.763213 10208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:35.763252 10208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:52:35.765331 10208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:39755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=42683
--tserver_master_addrs=127.22.30.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:35.765539 10208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:35.765775 10208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:35.771977 10268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:35.772187 10270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.772924 10208 server_base.cc:1034] running on GCE node
W20260808 22:52:35.773138 10267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.773383 10208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:35.773618 10208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:35.774791 10208 hybrid_clock.cc:648] HybridClock initialized: now 1786229555774755 us; error 47 us; skew 500 ppm
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:35.777782 10208 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:52:35.778276 10208 webserver.cc:460] Webserver started at http://127.22.30.1:42683/ using document root <none> and password file <none>
I20260808 22:52:35.778513 10208 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:35.778587 10208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:35.803783 10208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260808 22:52:35.804540 10286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.804693 10208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:35.804773 10208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "8b023fa1d71c421f8fafdfca04c01ad4"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "5ddd736d0e1b1196f548be2f507e5550"
server_key_iv: "60a687c71d5824daebf279a03d73bc9d"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.805217 10208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:35.824182 10208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:35.826416 10208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:35.826560 10208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:35.826776 10208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:35.827105 10208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:35.827152 10208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.827184 10208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:35.827204 10208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.833088 10208 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:39755
I20260808 22:52:35.833460 10208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:52:35.843129 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 10208
I20260808 22:52:35.843473 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:37273
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:45377
--webserver_port=35735
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:35.844784 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:39755 every 8 connection(s)
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:35.859826 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:35.845616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:45377 (local address 127.22.30.1:49045)
0808 22:52:35.850697 (+ 5081us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:35.850742 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:35.851354 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:35.851975 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:35.851979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:35.852200 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:35.852394 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:35.852398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.853494 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.853495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:35.854222 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:35.854224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.854782 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.854783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:35.854892 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:35.855424 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:35.855433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:35.856573 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:35.858282 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:35.858288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:35.858290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:35.858497 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:35.858734 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:35.858735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:35.858735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:35.858841 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:35.859213 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:35.859218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:35.859347 (+ 129us) client_negotiation.cc:769] Sending connection context
0808 22:52:35.859470 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 22:52:35.859626 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5022,"thread_start_us":96,"threads_started":1}
I20260808 22:52:35.860482 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:35.845883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:49045 (local address 127.22.30.62:45377)
0808 22:52:35.851583 (+ 5700us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:35.851589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:35.851613 (+ 24us) server_negotiation.cc:407] Connection header received
0808 22:52:35.851760 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:35.851766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:35.851809 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:35.851936 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:35.852518 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.853400 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.854312 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:35.854678 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:35.854804 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:35.856680 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:35.856684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:35.856684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:35.856693 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:35.858182 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:35.858577 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:35.858580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:35.858581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:35.858666 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:35.858905 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:35.858906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:35.858906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:35.859126 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:35.859218 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:35.860201 (+ 983us) server_negotiation.cc:299] Negotiation successful
0808 22:52:35.860314 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5647,"thread_start_us":83,"threads_started":1}
I20260808 22:52:35.861433 10462 heartbeater.cc:347] Connected to a master server at 127.22.30.62:45377
I20260808 22:52:35.861521 10462 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:35.861701 10462 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
W20260808 22:52:35.862540 10462 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:45377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 22:52:35.862625 10462 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:35.862689 10462 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:35.863070 10462 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:35.863147 10462 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
W20260808 22:52:35.863559 10462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:52:35.967447 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:35.967701 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:35.967743 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:35.969394 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:35.969462 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:35.969496 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:52:35.971481 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=35735
--tserver_master_addrs=127.22.30.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:35.971721 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:35.971958 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:35.977419 10475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:35.977449 10476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.977967 10466 server_base.cc:1034] running on GCE node
W20260808 22:52:35.978016 10478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:35.978264 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:35.978561 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:35.979719 10466 hybrid_clock.cc:648] HybridClock initialized: now 1786229555979694 us; error 41 us; skew 500 ppm
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:35.982547 10466 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:52:35.983062 10466 webserver.cc:460] Webserver started at http://127.22.30.2:35735/ using document root <none> and password file <none>
I20260808 22:52:35.983341 10466 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:35.983445 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:52:35 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:35.997771 10466 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:35.998518 10490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:35.998711 10466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:35.998800 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "ffa4d7f957744efabf65df8ae2329646"
format_stamp: "Formatted at 2026-08-08 22:52:34 on dist-test-slave-x4qh"
server_key: "508ebf9452a49386d7cf1c75170a2d42"
server_key_iv: "82e77bdc1a143a3ba03734478627ed3a"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:35.999164 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:36.017305 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:36.019505 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:36.019677 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:36.019865 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:36.020144 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:36.020191 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:36.020231 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:36.020253 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:36.027025 10466 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:37273
I20260808 22:52:36.027467 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:52:36.027501 10654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:37273 every 8 connection(s)
I20260808 22:52:36.029662 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 10466
I20260808 22:52:36.029950 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.30.20:57607/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:46599
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:45377
--webserver_port=37155
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:52:36 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229555, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:36.037865 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:36.028227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:33879 (local address 127.22.30.62:45377)
0808 22:52:36.028279 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:36.028281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:36.028530 (+ 249us) server_negotiation.cc:407] Connection header received
0808 22:52:36.029138 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:36.029139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:36.029154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:36.029215 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:36.029967 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.030744 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.032205 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.032388 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.032483 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:36.034421 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:36.034422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:36.034423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:36.034430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:36.036091 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:36.036733 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:36.036735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:36.036735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:36.036800 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:36.037109 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:36.037111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:36.037111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:36.037247 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:36.037339 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:36.037702 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 22:52:36.037782 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:52:36.038332 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:36.028208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:45377 (local address 127.22.30.2:33879)
0808 22:52:36.028388 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:36.028431 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:36.029012 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:36.029347 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:36.029352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:36.029580 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:36.029850 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:36.029857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.030862 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.030862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:36.032118 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:36.032121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.032503 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.032504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:36.032622 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:36.033050 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:36.033058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:36.034325 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:36.036170 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:36.036177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:36.036179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:36.036525 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:36.036878 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:36.036879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:36.036880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:36.037025 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:36.037339 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:36.037346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:36.037525 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 22:52:36.037647 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 22:52:36.037809 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260808 22:52:36.039085 10657 heartbeater.cc:347] Connected to a master server at 127.22.30.62:45377
I20260808 22:52:36.039197 10657 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.039414 10657 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
W20260808 22:52:36.040058 10657 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:45377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 22:52:36.040127 10657 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.040184 10657 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:36.040539 10657 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.040616 10657 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
W20260808 22:52:36.040943 10657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:52:36.133023 10663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:36.133265 10663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:36.133301 10663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:36.134756 10663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:36.134807 10663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:36.134845 10663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:52:36.136766 10663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.30.20:57607/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:46599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=37155
--tserver_master_addrs=127.22.30.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:36.137017 10663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:36.137295 10663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:36.143404 10671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:36.143661 10676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:36.143783 10663 server_base.cc:1034] running on GCE node
W20260808 22:52:36.143918 10672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:36.144129 10663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:36.144384 10663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:36.145539 10663 hybrid_clock.cc:648] HybridClock initialized: now 1786229556145529 us; error 28 us; skew 500 ppm
Aug 08 22:52:36 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229556, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:36.148092 10663 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:52:36.148576 10663 webserver.cc:460] Webserver started at http://127.22.30.3:37155/ using document root <none> and password file <none>
I20260808 22:52:36.148816 10663 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:36.148898 10663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 22:52:36 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229556, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:36.167682 10663 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260808 22:52:36.168373 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:36.168547 10663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:52:36.168622 10663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "c8b480c38c9d423e92fbdccbab9defa0"
format_stamp: "Formatted at 2026-08-08 22:52:35 on dist-test-slave-x4qh"
server_key: "427ed8e952e5a0bd01a5dcb60ea96d75"
server_key_iv: "e8070b4de5e9881d7e85cdf5c4114fa6"
server_key_version: "kuduclusterkey@0"
I20260808 22:52:36.169006 10663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:36.186748 10663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:36.189030 10663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:36.189231 10663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:36.189455 10663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:36.189800 10663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:36.189883 10663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:36.189954 10663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:36.190007 10663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:36.196631 10663 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:46599
I20260808 22:52:36.196717 10861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:46599 every 8 connection(s)
I20260808 22:52:36.197108 10663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:52:36.203959 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 10663
Aug 08 22:52:36 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229556, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:36.208907 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:36.198070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:52483 (local address 127.22.30.62:45377)
0808 22:52:36.198132 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:36.198135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:36.198360 (+ 225us) server_negotiation.cc:407] Connection header received
0808 22:52:36.199939 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:36.199941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:36.199953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:36.200028 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:36.200665 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.201398 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.202746 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.203031 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.203107 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:36.205763 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:36.205764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:36.205765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:36.205771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:36.207353 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:36.207819 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:36.207820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:36.207821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:36.207874 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:36.208163 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:36.208164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:36.208164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:36.208295 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:36.208334 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:36.208697 (+ 363us) server_negotiation.cc:299] Negotiation successful
0808 22:52:36.208781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:52:36.209656 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:36.198053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:45377 (local address 127.22.30.3:52483)
0808 22:52:36.198219 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:36.198263 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:36.199815 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:36.200106 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:36.200109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:36.200334 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:36.200562 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:36.200567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.201502 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.201503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:36.202636 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:36.202639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.203135 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.203137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:36.203273 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:36.204184 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:36.204195 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:36.205664 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:36.207463 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:36.207470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:36.207471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:36.207734 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:36.207939 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:36.207939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:36.207939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:36.208069 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:36.208379 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:36.208385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:36.208547 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 22:52:36.209254 (+ 707us) client_negotiation.cc:240] Negotiation successful
0808 22:52:36.209420 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260808 22:52:36.210328 10863 heartbeater.cc:347] Connected to a master server at 127.22.30.62:45377
I20260808 22:52:36.210409 10863 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.210578 10863 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
W20260808 22:52:36.211193 10863 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:45377 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 22:52:36.211254 10863 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.211333 10863 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:36.211683 10863 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.211752 10863 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
W20260808 22:52:36.212052 10863 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:52:36.213025 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:36.204479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.22.30.62:45377)
0808 22:52:36.204743 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:36.204745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:36.204752 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:52:36.204784 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:36.204785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:36.204795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:36.205494 (+ 699us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:36.206293 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.207025 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.208168 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:36.208453 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:36.208508 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:36.210260 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:36.210260 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:36.210261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:36.210267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:36.211934 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:36.212294 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:36.212296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:36.212296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:36.212348 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:36.212547 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:36.212548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:36.212548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:36.212670 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:36.212812 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:36.212849 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 22:52:36.212928 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":43,"threads_started":1}
Aug 08 22:52:36 dist-test-slave-x4qh krb5kdc[8585](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786229556, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:36.863759 10462 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:36.863911 10462 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:36.864240 10146 ts_manager.cc:194] Registered new tserver with Master: 8b023fa1d71c421f8fafdfca04c01ad4 (127.22.30.1:39755)
I20260808 22:52:36.865291 10146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:49045
I20260808 22:52:37.041131 10657 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:37.041265 10657 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:37.041522 10146 ts_manager.cc:194] Registered new tserver with Master: ffa4d7f957744efabf65df8ae2329646 (127.22.30.2:37273)
I20260808 22:52:37.042054 10146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:33879
Aug 08 22:52:37 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229556, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
I20260808 22:52:37.212219 10863 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:37.212370 10863 heartbeater.cc:511] Master 127.22.30.62:45377 requested a full tablet report, sending...
I20260808 22:52:37.212661 10146 ts_manager.cc:194] Registered new tserver with Master: c8b480c38c9d423e92fbdccbab9defa0 (127.22.30.3:46599)
I20260808 22:52:37.213137 10146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:52483
I20260808 22:52:37.221108 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:52:37.230437 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.222098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.22.30.62:45377)
0808 22:52:37.222214 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.222216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.222244 (+ 28us) server_negotiation.cc:407] Connection header received
0808 22:52:37.222308 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.222308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.222320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.222395 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:37.223057 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.223797 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.224851 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.225160 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.225242 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.226762 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:37.226763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:37.226764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:37.226770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:37.228907 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:37.229333 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:37.229334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:37.229335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:37.229396 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:37.229617 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:37.229618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:37.229618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:37.229746 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:37.229836 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.230268 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.230346 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260808 22:52:37.232738 10146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:52:37.234987 10146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:52:37.251194 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.241889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:46599 (local address 127.0.0.1:56320)
0808 22:52:37.243291 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.243327 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:37.243807 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:37.245011 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:37.245012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:37.245014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:37.245074 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.245076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.246246 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.246247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.249654 (+ 3407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.249657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.250916 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.250917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.251000 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.251013 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 22:52:37.251077 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 22:52:37.251124 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"thread_start_us":93,"threads_started":1}
I20260808 22:52:37.251606 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.242419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.22.30.3:46599)
0808 22:52:37.243164 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.243168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.243420 (+ 252us) server_negotiation.cc:407] Connection header received
0808 22:52:37.243888 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.243890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.243903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.243963 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:37.245154 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.246146 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.249746 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.250714 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.250799 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.250867 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.251494 (+ 627us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.251536 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":54,"threads_started":1}
I20260808 22:52:37.251699 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.242920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.22.30.1:39755)
0808 22:52:37.247935 (+ 5015us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.247940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.247951 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:52:37.247982 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.247984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.247998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.248081 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:37.248343 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.249219 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.250365 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.251069 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.251160 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.251231 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.251589 (+ 358us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.251630 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4978,"thread_start_us":54,"threads_started":1}
I20260808 22:52:37.251523 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.242830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39755 (local address 127.0.0.1:43500)
0808 22:52:37.247330 (+ 4500us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.247338 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:37.247399 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:37.248190 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:37.248191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:37.248193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:37.248253 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.248254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.249322 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.249323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.250274 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.250277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.251266 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.251267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.251333 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.251337 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:52:37.251424 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 22:52:37.251460 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4484,"thread_start_us":69,"threads_started":1}
I20260808 22:52:37.252242 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.242132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37273 (local address 127.0.0.1:51192)
0808 22:52:37.243473 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.243481 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:37.243535 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:37.244032 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:37.244035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:37.244037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:37.244092 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.244094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.248005 (+ 3911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.248006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.249134 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.249138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.252017 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.252018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.252077 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.252081 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:52:37.252147 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:52:37.252181 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"thread_start_us":63,"threads_started":1}
I20260808 22:52:37.252480 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.242551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51192 (local address 127.22.30.2:37273)
0808 22:52:37.243230 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.243233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.243611 (+ 378us) server_negotiation.cc:407] Connection header received
0808 22:52:37.243657 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.243659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.243675 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.243737 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:37.244170 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.247912 (+ 3742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.249241 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.249930 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.250004 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.250066 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.252370 (+ 2304us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.252411 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":44,"threads_started":1}
I20260808 22:52:37.252511 10342 tablet_service.cc:1467] Processing CreateTablet for tablet 99f082f4089c4b41bce39f9670429acd (DEFAULT_TABLE table=test-table [id=ba62832eb4c04f56bbaa7bec6977ebae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:37.252892 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f082f4089c4b41bce39f9670429acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:37.255913 10915 tablet_bootstrap.cc:492] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: Bootstrap starting.
I20260808 22:52:37.255995 10562 tablet_service.cc:1467] Processing CreateTablet for tablet 99f082f4089c4b41bce39f9670429acd (DEFAULT_TABLE table=test-table [id=ba62832eb4c04f56bbaa7bec6977ebae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:37.256300 10562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f082f4089c4b41bce39f9670429acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:37.256546 10915 tablet_bootstrap.cc:654] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:37.256822 10915 log.cc:824] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:37.256824 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 99f082f4089c4b41bce39f9670429acd (DEFAULT_TABLE table=test-table [id=ba62832eb4c04f56bbaa7bec6977ebae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:37.257169 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f082f4089c4b41bce39f9670429acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:37.259516 10919 tablet_bootstrap.cc:492] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Bootstrap starting.
I20260808 22:52:37.259773 10921 tablet_bootstrap.cc:492] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Bootstrap starting.
I20260808 22:52:37.260356 10919 tablet_bootstrap.cc:654] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:37.260365 10921 tablet_bootstrap.cc:654] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:37.260675 10919 log.cc:824] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:37.260681 10921 log.cc:824] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:37.261550 10921 tablet_bootstrap.cc:492] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: No bootstrap required, opened a new log
I20260808 22:52:37.261621 10921 ts_tablet_manager.cc:1397] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:52:37.261880 10915 tablet_bootstrap.cc:492] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: No bootstrap required, opened a new log
I20260808 22:52:37.261936 10915 ts_tablet_manager.cc:1397] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260808 22:52:37.264159 10921 raft_consensus.cc:348] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.264357 10921 raft_consensus.cc:374] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:37.264417 10921 raft_consensus.cc:729] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b480c38c9d423e92fbdccbab9defa0, State: Initialized, Role: FOLLOWER
I20260808 22:52:37.264303 10915 raft_consensus.cc:348] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.264503 10915 raft_consensus.cc:374] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:37.264566 10915 raft_consensus.cc:729] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b023fa1d71c421f8fafdfca04c01ad4, State: Initialized, Role: FOLLOWER
I20260808 22:52:37.264554 10921 consensus_queue.cc:260] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.264703 10915 consensus_queue.cc:260] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.264873 10921 ts_tablet_manager.cc:1428] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:52:37.264959 10915 ts_tablet_manager.cc:1428] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:52:37.265038 10863 heartbeater.cc:503] Master 127.22.30.62:45377 was elected leader, sending a full tablet report...
I20260808 22:52:37.265136 10462 heartbeater.cc:503] Master 127.22.30.62:45377 was elected leader, sending a full tablet report...
I20260808 22:52:37.266352 10919 tablet_bootstrap.cc:492] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: No bootstrap required, opened a new log
I20260808 22:52:37.266425 10919 ts_tablet_manager.cc:1397] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260808 22:52:37.268687 10919 raft_consensus.cc:348] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.268924 10919 raft_consensus.cc:374] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:37.269002 10919 raft_consensus.cc:729] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa4d7f957744efabf65df8ae2329646, State: Initialized, Role: FOLLOWER
I20260808 22:52:37.269171 10919 consensus_queue.cc:260] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.269479 10919 ts_tablet_manager.cc:1428] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:52:37.269609 10657 heartbeater.cc:503] Master 127.22.30.62:45377 was elected leader, sending a full tablet report...
W20260808 22:52:37.278767 10658 tablet.cc:2369] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 22:52:37 dist-test-slave-x4qh krb5kdc[8585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229556, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for HTTP/127.22.30.20@KRBTEST.COM
W20260808 22:52:37.346166 10464 tablet.cc:2369] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:52:37.394729 10928 raft_consensus.cc:482] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:37.394905 10928 raft_consensus.cc:504] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.395280 10928 leader_election.cc:288] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffa4d7f957744efabf65df8ae2329646 (127.22.30.2:37273), 8b023fa1d71c421f8fafdfca04c01ad4 (127.22.30.1:39755)
I20260808 22:52:37.399873 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.395971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:39535 (local address 127.22.30.1:39755)
0808 22:52:37.396025 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.396027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.396253 (+ 226us) server_negotiation.cc:407] Connection header received
0808 22:52:37.396350 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.396351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.396363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.396390 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:37.396604 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.397280 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.398753 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.399355 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.399538 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.399551 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.399765 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.399807 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:52:37.400309 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.395741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39755 (local address 127.22.30.3:39535)
0808 22:52:37.396188 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.396199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:37.396279 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:37.396451 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:37.396452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:37.396453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:37.396539 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.396541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.397384 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.397385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.398617 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.398621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.399609 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.399609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.399690 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.399693 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:52:37.400158 (+ 465us) client_negotiation.cc:240] Negotiation successful
0808 22:52:37.400202 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":107,"threads_started":1}
I20260808 22:52:37.400890 10389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f082f4089c4b41bce39f9670429acd" candidate_uuid: "c8b480c38c9d423e92fbdccbab9defa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" is_pre_election: true
I20260808 22:52:37.401077 10389 raft_consensus.cc:2454] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b480c38c9d423e92fbdccbab9defa0 in term 0.
I20260808 22:52:37.401320 10694 leader_election.cc:302] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b023fa1d71c421f8fafdfca04c01ad4, c8b480c38c9d423e92fbdccbab9defa0; no voters:
I20260808 22:52:37.401503 10928 raft_consensus.cc:2781] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:52:37.401556 10928 raft_consensus.cc:482] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:52:37.401580 10928 raft_consensus.cc:3037] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:37.402580 10928 raft_consensus.cc:504] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.402699 10928 leader_election.cc:288] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [CANDIDATE]: Term 1 election: Requested vote from peers ffa4d7f957744efabf65df8ae2329646 (127.22.30.2:37273), 8b023fa1d71c421f8fafdfca04c01ad4 (127.22.30.1:39755)
I20260808 22:52:37.403008 10389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f082f4089c4b41bce39f9670429acd" candidate_uuid: "c8b480c38c9d423e92fbdccbab9defa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b023fa1d71c421f8fafdfca04c01ad4"
I20260808 22:52:37.403079 10389 raft_consensus.cc:3037] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:37.404066 10389 raft_consensus.cc:2454] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b480c38c9d423e92fbdccbab9defa0 in term 1.
I20260808 22:52:37.404084 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.395487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:33197 (local address 127.22.30.2:37273)
0808 22:52:37.395532 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.395534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.400842 (+ 5308us) server_negotiation.cc:407] Connection header received
0808 22:52:37.400945 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.400946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.400960 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.400990 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:37.401184 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.401874 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.403067 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.403697 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.403743 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.403752 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.403971 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.404010 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 22:52:37.404289 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.396021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37273 (local address 127.22.30.3:33197)
0808 22:52:37.400791 (+ 4770us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.400802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:37.400870 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:37.401054 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:37.401055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:37.401056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:37.401084 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.401085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.401978 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.401979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.402969 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:37.402972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.403808 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.403808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:37.403869 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.403872 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:52:37.404199 (+ 327us) client_negotiation.cc:240] Negotiation successful
0808 22:52:37.404234 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4757,"thread_start_us":94,"threads_started":1}
I20260808 22:52:37.404706 10590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f082f4089c4b41bce39f9670429acd" candidate_uuid: "c8b480c38c9d423e92fbdccbab9defa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa4d7f957744efabf65df8ae2329646" is_pre_election: true
I20260808 22:52:37.404844 10590 raft_consensus.cc:2454] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b480c38c9d423e92fbdccbab9defa0 in term 0.
I20260808 22:52:37.404963 10588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f082f4089c4b41bce39f9670429acd" candidate_uuid: "c8b480c38c9d423e92fbdccbab9defa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa4d7f957744efabf65df8ae2329646"
I20260808 22:52:37.405031 10588 raft_consensus.cc:3037] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:37.405092 10694 leader_election.cc:302] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b023fa1d71c421f8fafdfca04c01ad4, c8b480c38c9d423e92fbdccbab9defa0; no voters:
I20260808 22:52:37.405298 10928 raft_consensus.cc:2781] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:37.405436 10928 raft_consensus.cc:686] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 1 LEADER]: Becoming Leader. State: Replica: c8b480c38c9d423e92fbdccbab9defa0, State: Running, Role: LEADER
I20260808 22:52:37.405514 10928 consensus_queue.cc:237] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } }
I20260808 22:52:37.405913 10588 raft_consensus.cc:2454] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b480c38c9d423e92fbdccbab9defa0 in term 1.
I20260808 22:52:37.406381 10145 catalog_manager.cc:5495] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8b480c38c9d423e92fbdccbab9defa0 (127.22.30.3). New cstate: current_term: 1 leader_uuid: "c8b480c38c9d423e92fbdccbab9defa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b480c38c9d423e92fbdccbab9defa0" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20260808 22:52:37.428548 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.425468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.22.30.3:46599)
0808 22:52:37.425530 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:37.425532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:37.425541 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:52:37.425571 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:37.425572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:37.425589 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:37.425678 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:52:37.425836 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.426451 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.427478 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:37.427728 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:37.427793 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:37.428001 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:52:37.428133 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:52:37.428212 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:37.428445 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 22:52:37.428465 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:52:37.430418 10389 raft_consensus.cc:1261] T 99f082f4089c4b41bce39f9670429acd P 8b023fa1d71c421f8fafdfca04c01ad4 [term 1 FOLLOWER]: Refusing update from remote peer c8b480c38c9d423e92fbdccbab9defa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:37.430447 10588 raft_consensus.cc:1261] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 1 FOLLOWER]: Refusing update from remote peer c8b480c38c9d423e92fbdccbab9defa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:37.430765 10928 consensus_queue.cc:1035] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa4d7f957744efabf65df8ae2329646" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:37.431083 10936 consensus_queue.cc:1035] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b023fa1d71c421f8fafdfca04c01ad4" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:37.435397 10947 mvcc.cc:205] Tried to move back new op lower bound from 7316396267232301056 to 7316396267133542400. Current Snapshot: MvccSnapshot[applied={T|T < 7316396267232301056}]
I20260808 22:52:37.436111 10943 mvcc.cc:205] Tried to move back new op lower bound from 7316396267232301056 to 7316396267133542400. Current Snapshot: MvccSnapshot[applied={T|T < 7316396267232301056}]
I20260808 22:52:37.436570 10945 mvcc.cc:205] Tried to move back new op lower bound from 7316396267232301056 to 7316396267133542400. Current Snapshot: MvccSnapshot[applied={T|T < 7316396267232301056}]
I20260808 22:52:37.438813 10146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:52:37.438896 10146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:52:37.439630 10146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a9673005d2743e982c7783c685f08fb: Sending DeleteTablet for 3 replicas of tablet 99f082f4089c4b41bce39f9670429acd
I20260808 22:52:37.439953 10562 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f082f4089c4b41bce39f9670429acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:37 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:51192
I20260808 22:52:37.440357 10342 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f082f4089c4b41bce39f9670429acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:37 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:43500
I20260808 22:52:37.440781 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f082f4089c4b41bce39f9670429acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:37 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56320
I20260808 22:52:37.441222 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 10208
I20260808 22:52:37.451149 10958 tablet_replica.cc:330] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: stopping tablet replica
I20260808 22:52:37.451253 10958 raft_consensus.cc:2229] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:52:37.451359 10958 raft_consensus.cc:2258] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:37.456483 10958 ts_tablet_manager.cc:1905] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:37.458256 10958 ts_tablet_manager.cc:1918] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:52:37.458391 10958 log.cc:1196] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/99f082f4089c4b41bce39f9670429acd
I20260808 22:52:37.458808 10958 ts_tablet_manager.cc:1939] T 99f082f4089c4b41bce39f9670429acd P c8b480c38c9d423e92fbdccbab9defa0: Deleting consensus metadata
I20260808 22:52:37.459257 10123 catalog_manager.cc:4841] TS c8b480c38c9d423e92fbdccbab9defa0 (127.22.30.3:46599): tablet 99f082f4089c4b41bce39f9670429acd (table test-table [id=ba62832eb4c04f56bbaa7bec6977ebae]) successfully deleted
I20260808 22:52:37.459486 10956 tablet_replica.cc:330] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: stopping tablet replica
W20260808 22:52:37.459551 10127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:39755 (error 108)
I20260808 22:52:37.459590 10956 raft_consensus.cc:2229] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:52:37.459690 10956 raft_consensus.cc:2258] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:37.460129 10956 ts_tablet_manager.cc:1905] T 99f082f4089c4b41bce39f9670429acd P ffa4d7f957744efabf65df8ae2329646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:37.460383 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 10466
I20260808 22:52:37.501009 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:37.500850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:39755 (local address 127.0.0.1:43506)
0808 22:52:37.500908 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.500965 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 22:52:37.502048 10127 catalog_manager.cc:4584] TS 8b023fa1d71c421f8fafdfca04c01ad4 (127.22.30.1:39755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99f082f4089c4b41bce39f9670429acd: Network error: Client connection negotiation failed: client connection to 127.22.30.1:39755: connect: Connection refused (error 111)
W20260808 22:52:37.507669 10907 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:52:37.507365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37273 (local address 127.0.0.1:51204)
0808 22:52:37.507423 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:37.507434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:37.507478 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:37.507615 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.2:37273: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 22:52:37.508309 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 10663
I20260808 22:52:37.520232 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 10092
I20260808 22:52:37.531682 22648 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 22:52:37.876130 22648 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 22:52:37.876219 22648 mini_ranger.cc:66] Stopping Ranger...
I20260808 22:52:38.220064 22648 mini_ranger.cc:68] Stopped Ranger
2026-08-08 22:52:38.220 UTC [8637] LOG: received smart shutdown request
2026-08-08 22:52:38.224 UTC [8637] LOG: background worker "logical replication launcher" (PID 8652) exited with exit code 1
2026-08-08 22:52:38.224 UTC [8645] LOG: shutting down
2026-08-08 22:52:38.364 UTC [8637] LOG: database system is shut down
2026-08-08T22:52:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45322 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:52:38 dist-test-slave-x4qh krb5kdc[10985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:52:38 dist-test-slave-x4qh krb5kdc[10985](info): set up 2 sockets
Aug 08 22:52:38 dist-test-slave-x4qh krb5kdc[10985](info): commencing operation
krb5kdc: starting...
W20260808 22:52:39.623003 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:52:39 dist-test-slave-x4qh krb5kdc[10985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:52:39.734354 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43515
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43515,127.22.30.61:38485,127.22.30.60:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:39.819914 11009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:39.820114 11009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:39.820148 11009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:39.821211 11009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:39.821274 11009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:39.821297 11009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:39.821317 11009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:39.821349 11009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:39.822690 11009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43515,127.22.30.61:38485,127.22.30.60:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:39.822932 11009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:39.823122 11009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:39.827826 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:39.827868 11016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:39.827988 11009 server_base.cc:1034] running on GCE node
W20260808 22:52:39.828029 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:39.828261 11009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:39.828567 11009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:39.829731 11009 hybrid_clock.cc:648] HybridClock initialized: now 1786229559829712 us; error 35 us; skew 500 ppm
Aug 08 22:52:39 dist-test-slave-x4qh krb5kdc[10985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:39.832446 11009 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:39.833033 11009 webserver.cc:460] Webserver started at http://127.22.30.62:41087/ using document root <none> and password file <none>
I20260808 22:52:39.833323 11009 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:39.833396 11009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:39.833542 11009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:39.835083 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "0f792c7980b546049b7045a7a189036d"
format_stamp: "Formatted at 2026-08-08 22:52:39 on dist-test-slave-x4qh"
server_key: "6a9838fe2c3a950e35903c05a25f6c41"
server_key_iv: "0157366b0f51d90983fb0aa4bdbe8103"
server_key_version: "encryptionkey@0"
I20260808 22:52:39.835567 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "0f792c7980b546049b7045a7a189036d"
format_stamp: "Formatted at 2026-08-08 22:52:39 on dist-test-slave-x4qh"
server_key: "6a9838fe2c3a950e35903c05a25f6c41"
server_key_iv: "0157366b0f51d90983fb0aa4bdbe8103"
server_key_version: "encryptionkey@0"
I20260808 22:52:39.837569 11009 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 22:52:39.838517 11032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:39.838725 11009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:39.838830 11009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "0f792c7980b546049b7045a7a189036d"
format_stamp: "Formatted at 2026-08-08 22:52:39 on dist-test-slave-x4qh"
server_key: "6a9838fe2c3a950e35903c05a25f6c41"
server_key_iv: "0157366b0f51d90983fb0aa4bdbe8103"
server_key_version: "encryptionkey@0"
I20260808 22:52:39.838917 11009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:39.854364 11009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:39.856448 11009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:39.856565 11009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:39.860838 11009 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:43515
I20260808 22:52:39.860888 11102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:43515 every 8 connection(s)
I20260808 22:52:39.861214 11009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:39.862296 11106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:39.864822 11106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:52:39.871032 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11009
I20260808 22:52:39.871148 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:52:39.871332 22648 external_mini_cluster.cc:1428] Setting key 40b212d40610bf241fba162f8875466b
Aug 08 22:52:39 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:39.875974 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.865603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.22.30.62:43515)
0808 22:52:39.865778 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:39.865789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:39.865869 (+ 80us) server_negotiation.cc:407] Connection header received
0808 22:52:39.866388 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:39.866391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:39.866414 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:39.866519 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:39.866997 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:39.868498 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:39.869327 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:39.869917 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:39.870040 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:39.872464 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:39.872473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:39.872473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:39.872480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:39.874160 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:39.874542 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:39.874544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:39.874545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:39.874623 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:39.874882 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:39.874883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:39.874883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:39.875198 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:39.875297 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:39.875529 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 22:52:39.875681 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260808 22:52:39.875978 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.865548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43515 (local address 127.0.0.1:34584)
0808 22:52:39.865762 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.865800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:39.866232 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:39.866543 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:39.866545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:39.866761 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:39.866904 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:39.866907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:39.868595 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:39.868596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:39.869241 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:39.869243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:39.870009 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:39.870009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:39.870065 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:39.870531 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:39.870539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:39.872366 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:39.874255 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:39.874261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:39.874263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:39.874459 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:39.874689 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:39.874690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:39.874690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:39.874797 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:39.875288 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:39.875296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:39.875388 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 22:52:39.875486 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 22:52:39.875774 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":126,"thread_start_us":92,"threads_started":1}
I20260808 22:52:39.876637 11106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:52:39.877038 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.876912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52796)
0808 22:52:39.876962 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.877010 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 22:52:39.877158 11034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
I20260808 22:52:39.877524 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.877435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52808)
0808 22:52:39.877480 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.877504 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 22:52:39.877625 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:38485: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.22.30.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab.
Entry for principal kudu/127.22.30.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab.
I20260808 22:52:39.900341 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:38485
--webserver_interface=127.22.30.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab
--principal=kudu/127.22.30.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43515,127.22.30.61:38485,127.22.30.60:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:39.926844 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } attempt: 1
I20260808 22:52:39.927408 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.927228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52824)
0808 22:52:39.927307 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.927349 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 22:52:39.927843 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.927734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52840)
0808 22:52:39.927793 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.927821 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260808 22:52:39.927980 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:38485: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
W20260808 22:52:39.984448 11118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:39.984613 11118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:39.984633 11118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:39.985647 11118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:39.985683 11118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:39.985697 11118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:39.985708 11118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:39.985754 11118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:39.987043 11118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43515,127.22.30.61:38485,127.22.30.60:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab
--principal=kudu/127.22.30.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.30.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:39.987258 11118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:39.987493 11118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:39.991964 11130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:39.991986 11128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:39.992113 11118 server_base.cc:1034] running on GCE node
W20260808 22:52:39.991973 11132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:39.992480 11118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:39.992751 11118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:39.993906 11118 hybrid_clock.cc:648] HybridClock initialized: now 1786229559993904 us; error 34 us; skew 500 ppm
Aug 08 22:52:39 dist-test-slave-x4qh krb5kdc[10985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:39.996116 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } attempt: 2
I20260808 22:52:39.996340 11118 init.cc:377] Logged in from keytab as kudu/127.22.30.61@KRBTEST.COM (short username kudu)
I20260808 22:52:39.996495 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.996343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52846)
0808 22:52:39.996417 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.996458 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:52:39.996790 11118 webserver.cc:460] Webserver started at http://127.22.30.61:46879/ using document root <none> and password file <none>
I20260808 22:52:39.996872 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:39.996742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52852)
0808 22:52:39.996815 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:39.996846 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 22:52:39.996973 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:38485: Network error: Client connection negotiation failed: client connection to 127.22.30.61:38485: connect: Connection refused (error 111)
I20260808 22:52:39.997032 11118 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:39.997087 11118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:39.997169 11118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:39.998034 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data/instance:
uuid: "2734124472bc4d2aaf5aaad526dd5248"
format_stamp: "Formatted at 2026-08-08 22:52:39 on dist-test-slave-x4qh"
server_key: "ca146ba64800585f69f62f318fbd5f80"
server_key_iv: "372a88753678639b8acd8659afe6ae54"
server_key_version: "encryptionkey@0"
I20260808 22:52:39.998312 11118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal/instance:
uuid: "2734124472bc4d2aaf5aaad526dd5248"
format_stamp: "Formatted at 2026-08-08 22:52:39 on dist-test-slave-x4qh"
server_key: "ca146ba64800585f69f62f318fbd5f80"
server_key_iv: "372a88753678639b8acd8659afe6ae54"
server_key_version: "encryptionkey@0"
I20260808 22:52:39.999596 11118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:52:40.000191 11142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:40.000365 11118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:40.000428 11118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal
uuid: "2734124472bc4d2aaf5aaad526dd5248"
format_stamp: "Formatted at 2026-08-08 22:52:39 on dist-test-slave-x4qh"
server_key: "ca146ba64800585f69f62f318fbd5f80"
server_key_iv: "372a88753678639b8acd8659afe6ae54"
server_key_version: "encryptionkey@0"
I20260808 22:52:40.000474 11118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:40.009758 11118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:40.011160 11118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:40.011250 11118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:40.015219 11118 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.61:38485
I20260808 22:52:40.015266 11219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.61:38485 every 8 connection(s)
I20260808 22:52:40.015578 11118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
I20260808 22:52:40.016388 11220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:40.017786 11220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:40.024981 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11118
I20260808 22:52:40.025135 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-1/wal/instance
I20260808 22:52:40.025332 22648 external_mini_cluster.cc:1428] Setting key e03e418c622a727543dc051ba59775aa
I20260808 22:52:40.026058 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.018512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.22.30.62:43515)
0808 22:52:40.018552 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.018554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.018709 (+ 155us) server_negotiation.cc:407] Connection header received
0808 22:52:40.019325 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.019326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.019337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.019429 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.020032 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.020468 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.021344 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.021520 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.021566 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.023300 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.023302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.023302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.023308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.024681 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.025129 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.025131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.025131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.025196 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.025508 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.025509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.025509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.025607 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.025654 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.025941 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.026004 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:52:40.026326 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.018417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43515 (local address 127.0.0.1:34586)
0808 22:52:40.018586 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.018623 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.019212 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.019556 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.019558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.019775 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.019961 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.019965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.020541 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.020542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.021261 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.021263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.021586 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.021587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.021676 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.022015 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.022022 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.023223 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.024761 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.024767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.024768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.025005 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.025274 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.025275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.025276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.025418 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.025670 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.025677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.025819 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.025920 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.026078 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260808 22:52:40.026777 11220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.30.60@KRBTEST.COM; defaulting to no policy
I20260808 22:52:40.033139 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.026994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52858)
0808 22:52:40.027035 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.027048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.027090 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.027308 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.027309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.027458 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.027497 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.027498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.028239 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.028240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.028712 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.028713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.029160 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.029161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.029224 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.029714 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.029718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.030661 (+ 943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.032132 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.032133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.032133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.032282 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.032507 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.032508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.032508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.032559 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.032829 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.032829 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.032865 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.032935 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.033026 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 22:52:40.033232 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.027030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.22.30.61:38485)
0808 22:52:40.027155 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.027158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.027175 (+ 17us) server_negotiation.cc:407] Connection header received
0808 22:52:40.027200 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.027201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.027211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.027304 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.027576 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.028163 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.028809 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.029087 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.029162 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.030734 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.030741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.030741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.030747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.032044 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.032386 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.032391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.032391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.032441 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.032638 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.032639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.032639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.032758 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.032846 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.032961 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.033089 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260808 22:52:40.033712 11220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:52:40.034032 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.033928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50588)
0808 22:52:40.033979 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.034017 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 22:52:40.034173 11143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
I20260808 22:52:40.034518 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.034464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50592)
0808 22:52:40.034485 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.034505 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260808 22:52:40.034627 11220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:40261: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab.
Entry for principal kudu/127.22.30.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab.
I20260808 22:52:40.048029 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:40261
--webserver_interface=127.22.30.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab
--principal=kudu/127.22.30.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43515,127.22.30.61:38485,127.22.30.60:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:40.083841 11220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } attempt: 1
I20260808 22:52:40.084241 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.084113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50604)
0808 22:52:40.084165 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.084217 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 22:52:40.084685 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.084584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50606)
0808 22:52:40.084644 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.084669 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 22:52:40.084794 11220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:40261: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
I20260808 22:52:40.088112 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } attempt: 3
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:52:40.094295 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.088416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.22.30.61:38485)
0808 22:52:40.088470 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.088472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.088481 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:52:40.088569 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.088570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.088582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.088660 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.088970 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.089458 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.090089 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.090295 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.090367 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.092030 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.092031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.092032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.092038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.093239 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.093556 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.093557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.093557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.093608 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.093797 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.093798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.093798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.093906 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.093976 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.094116 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.094197 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:52:40.094295 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.088361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52868)
0808 22:52:40.088413 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.088425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.088491 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.088667 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.088668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.088831 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.088892 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.088894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.089549 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.089550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.090013 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.090014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.090366 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.090367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.090441 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.090887 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.090891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.091952 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.093319 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.093320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.093320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.093476 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.093675 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.093675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.093676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.093729 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.093970 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.093970 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.094012 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.094093 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.094196 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:52:40.094645 11106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:52:40.094954 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.094868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50616)
0808 22:52:40.094913 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.094937 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 22:52:40.095268 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.095184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50622)
0808 22:52:40.095229 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.095250 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:52:40.095355 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:40261: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
W20260808 22:52:40.123927 11232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:40.124132 11232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:40.124181 11232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:40.125274 11232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:40.125334 11232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:40.125358 11232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:40.125386 11232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:40.125408 11232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:40.126493 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } attempt: 1
I20260808 22:52:40.126775 11232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43515,127.22.30.61:38485,127.22.30.60:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab
--principal=kudu/127.22.30.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:40261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.30.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:40.126807 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.126708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50632)
0808 22:52:40.126756 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.126789 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 22:52:40.127004 11232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:40.127070 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.126992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50640)
0808 22:52:40.127028 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.127049 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 22:52:40.127205 11106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:40261: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
I20260808 22:52:40.127218 11232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:40.131598 11243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:40.131636 11242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:40.131624 11232 server_base.cc:1034] running on GCE node
W20260808 22:52:40.131606 11245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:40.131987 11232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:40.132261 11232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:40.133399 11232 hybrid_clock.cc:648] HybridClock initialized: now 1786229560133391 us; error 26 us; skew 500 ppm
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229560, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:40.135667 11232 init.cc:377] Logged in from keytab as kudu/127.22.30.60@KRBTEST.COM (short username kudu)
I20260808 22:52:40.136075 11232 webserver.cc:460] Webserver started at http://127.22.30.60:38795/ using document root <none> and password file <none>
I20260808 22:52:40.136291 11232 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:40.136330 11232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:40.136461 11232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:40.137316 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data/instance:
uuid: "388456b360b344ea855cda3778654a10"
format_stamp: "Formatted at 2026-08-08 22:52:40 on dist-test-slave-x4qh"
server_key: "b5be0f3c2c2138d119d725fd9d3354bf"
server_key_iv: "c377176f51a2007a21c8487824efbe54"
server_key_version: "encryptionkey@0"
I20260808 22:52:40.137622 11232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal/instance:
uuid: "388456b360b344ea855cda3778654a10"
format_stamp: "Formatted at 2026-08-08 22:52:40 on dist-test-slave-x4qh"
server_key: "b5be0f3c2c2138d119d725fd9d3354bf"
server_key_iv: "c377176f51a2007a21c8487824efbe54"
server_key_version: "encryptionkey@0"
I20260808 22:52:40.138813 11232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:52:40.139573 11257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:40.139751 11232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:40.139833 11232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal
uuid: "388456b360b344ea855cda3778654a10"
format_stamp: "Formatted at 2026-08-08 22:52:40 on dist-test-slave-x4qh"
server_key: "b5be0f3c2c2138d119d725fd9d3354bf"
server_key_iv: "c377176f51a2007a21c8487824efbe54"
server_key_version: "encryptionkey@0"
I20260808 22:52:40.139902 11232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:40.152927 11220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } attempt: 2
I20260808 22:52:40.153258 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.153151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50648)
0808 22:52:40.153214 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.153238 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 22:52:40.153609 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.153529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50656)
0808 22:52:40.153570 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.153593 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 22:52:40.153710 11220 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:40261: Network error: Client connection negotiation failed: client connection to 127.22.30.60:40261: connect: Connection refused (error 111)
I20260808 22:52:40.170899 11232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:40.173046 11232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:40.173205 11232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:40.176884 11232 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.60:40261
I20260808 22:52:40.176928 11342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.60:40261 every 8 connection(s)
I20260808 22:52:40.177274 11232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
I20260808 22:52:40.178043 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:40.179395 11343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:52:40.182595 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11232
I20260808 22:52:40.182713 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786229410580610-22648-0/minicluster-data/master-2/wal/instance
I20260808 22:52:40.182902 22648 external_mini_cluster.cc:1428] Setting key 9f942516060b12fb33fd0fd7b7197e95
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229560, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:40.188092 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.179971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.22.30.62:43515)
0808 22:52:40.180008 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.180010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.180181 (+ 171us) server_negotiation.cc:407] Connection header received
0808 22:52:40.180557 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.180558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.180566 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.180617 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.181179 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.181604 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.182474 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.182778 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.182853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.185343 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.185345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.185345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.185352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.186639 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.187092 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.187093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.187094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.187166 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.187447 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.187449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.187449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.187586 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.187689 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.187922 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.188008 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:52:40.188100 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.179920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43515 (local address 127.0.0.1:34598)
0808 22:52:40.180057 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.180093 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.180488 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.180677 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.180679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.180900 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.181086 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.181090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.181677 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.181678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.182368 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.182370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.182860 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.182861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.183001 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.183551 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.183560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.185235 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.186707 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.186711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.186713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.187019 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.187237 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.187238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.187238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.187353 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.187684 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.187688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.187792 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.187869 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.187968 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:52:40.189100 11343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:52:40.192523 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.183310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.22.30.60:40261)
0808 22:52:40.183479 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.183484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.183605 (+ 121us) server_negotiation.cc:407] Connection header received
0808 22:52:40.183680 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.183682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.183699 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.183757 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.184505 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.185460 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.186449 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.186744 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.186810 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.189527 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.189540 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.189541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.189547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.191044 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.191679 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.191682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.191683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.191744 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.191983 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.191984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.191984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.192103 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.192200 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.192352 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.192434 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229560, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:52:40.195526 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.189317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:38485 (local address 127.0.0.1:52880)
0808 22:52:40.189364 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.189374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.189414 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.189539 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.189540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.189697 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.189735 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.189736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.190613 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.190614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.191048 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.191049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.191432 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.191433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.191502 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.191865 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.191870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.193114 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.194646 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.194647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.194647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.194791 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.194969 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.194969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.194970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.195014 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.195242 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.195242 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.195276 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.195350 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.195443 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 22:52:40.195551 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.189397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.22.30.61:38485)
0808 22:52:40.189440 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.189442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.189449 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:52:40.189471 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.189472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.189480 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.189534 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.189818 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.190528 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.191135 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.191328 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.191433 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.193193 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.193194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.193195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.193202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.194574 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.194851 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.194852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.194853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.194906 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.195068 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.195069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.195069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.195186 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.195245 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.195402 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.195471 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:52:40.196002 11343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229560, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:52:40.200555 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.193318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.22.30.62:43515)
0808 22:52:40.193350 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.193352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.193359 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:52:40.193400 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.193405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.193413 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.193489 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.194048 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.194509 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.195300 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.195545 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.195617 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.197745 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.197747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.197747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.197754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.199469 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.199815 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.199816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.199816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.199860 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.200075 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.200076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.200076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.200174 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.200344 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.200384 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.200461 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:52:40.202325 11106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } attempt: 2
I20260808 22:52:40.203136 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.196222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50660)
0808 22:52:40.196316 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.196326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.196368 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.196475 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.196475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.196592 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.196613 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.196614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.197428 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.197429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.198033 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.198036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.198562 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.198564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.198626 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.198867 (+ 241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.198872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.200193 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.201956 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.201957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.201958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.202114 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.202511 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.202512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.202512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.202576 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.202858 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.202859 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.202893 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.202953 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.203039 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 22:52:40.203198 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.196292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.22.30.60:40261)
0808 22:52:40.196319 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.196321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.196355 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:52:40.196415 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.196416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.196424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.196472 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.196679 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.197341 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.198209 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.198473 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.198563 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.200295 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.200296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.200297 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.200303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.201901 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.202205 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.202206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.202206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.202253 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.202642 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.202642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.202643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.202761 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.202837 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.202971 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.203087 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":13}
I20260808 22:52:40.205286 11343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: Bootstrap starting.
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:52:40.206001 11343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:40.206364 11343 log.cc:824] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: Log is configured to *not* fsync() on all Append() calls
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:52:40.207296 11343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: No bootstrap required, opened a new log
I20260808 22:52:40.209390 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.201192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.22.30.61:38485)
0808 22:52:40.201228 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.201230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.201237 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:52:40.201300 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.201300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.201307 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.201347 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.201918 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.202448 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.203464 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.203687 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.203734 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.206044 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.206046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.206046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.206053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.207643 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.208373 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.208374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.208374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.208432 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.208683 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.208684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.208684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.208817 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.208972 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.209219 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.209291 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:52:40.209364 11343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.209492 11343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:40.209515 11343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388456b360b344ea855cda3778654a10, State: Initialized, Role: FOLLOWER
I20260808 22:52:40.209652 11343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.210048 11343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:40.210244 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.202571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50674)
0808 22:52:40.202628 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.202637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.202669 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.202897 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.202898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.203064 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.203114 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.203115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.204534 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.204536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.205269 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.205272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.205784 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.205785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.205843 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.206262 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.206267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.207513 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.209138 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.209139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.209139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.209354 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.209592 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.209593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.209593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.209654 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.209971 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.209972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.210012 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.210076 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.210166 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 22:52:40.210286 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.202626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.22.30.60:40261)
0808 22:52:40.202756 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.202758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.202764 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:52:40.202782 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.202783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.202792 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.202869 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.203735 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.204439 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.205361 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.205702 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.205784 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.207609 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.207610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.207610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.207616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.209028 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.209437 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.209439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.209439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.209501 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.209723 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.209724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.209724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.209883 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.209944 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.210122 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.210188 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260808 22:52:40.210978 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:52:40.211720 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.211818 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:52:40.211869 11371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:52:40.211915 11371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:52:40.211902 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: Bootstrap starting.
I20260808 22:52:40.212541 11106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:40.212819 11106 log.cc:824] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:40.213636 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: No bootstrap required, opened a new log
I20260808 22:52:40.212937 11232 master_runner.cc:186] Error getting master registration for 127.22.30.62:43515: OK, instance_id { permanent_uuid: "0f792c7980b546049b7045a7a189036d" instance_seqno: 1786229559859674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:52:40.214493 11232 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:52:40.215562 11106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.215668 11106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:40.215705 11106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f792c7980b546049b7045a7a189036d, State: Initialized, Role: FOLLOWER
I20260808 22:52:40.216048 11106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.216394 11106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:40.218681 11009 master_runner.cc:186] Error getting master registration for 127.22.30.61:38485: OK, instance_id { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" instance_seqno: 1786229560014132 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:52:40.222005 11009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 22:52:40.222268 11405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260808 22:52:40.222446 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.212704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.22.30.62:43515)
0808 22:52:40.212783 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.212785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.215537 (+ 2752us) server_negotiation.cc:407] Connection header received
0808 22:52:40.215598 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.215599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.215609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.215654 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.216341 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.216977 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.217928 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.218182 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.218214 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.219652 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.219653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.219654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.219660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.221250 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.221584 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.221585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.221585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.221632 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.221865 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.221866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.221866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.221975 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.222039 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.222245 (+ 206us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.222312 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260808 22:52:40.222354 11405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:52:40.226068 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.212750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.22.30.61:38485)
0808 22:52:40.212811 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.212812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.215794 (+ 2982us) server_negotiation.cc:407] Connection header received
0808 22:52:40.215875 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.215875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.215884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.215911 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.216513 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.217129 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.217971 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.218220 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.218252 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.222941 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.222942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.222942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.222949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.224470 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.225226 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.225227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.225227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.225285 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.225508 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.225509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.225509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.225623 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.225702 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.225912 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.225977 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:52:40.226321 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.226454 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:52:40.226938 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.212650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.22.30.60:40261)
0808 22:52:40.213200 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.213202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.215883 (+ 2681us) server_negotiation.cc:407] Connection header received
0808 22:52:40.215904 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.215905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.215915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.215967 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.219241 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.219983 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.220938 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.221238 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.221275 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.223717 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.223718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.223719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.223726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.225791 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.226129 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.226130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.226130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.226183 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.226399 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.226400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.226400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.226523 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.226585 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.226784 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.226861 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506}
I20260808 22:52:40.244835 11220 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } attempt: 3
Aug 08 22:52:40 dist-test-slave-x4qh krb5kdc[10985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229559, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:52:40.250936 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.245128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.22.30.60:40261)
0808 22:52:40.245212 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:40.245213 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:40.245218 (+ 5us) server_negotiation.cc:407] Connection header received
0808 22:52:40.245231 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:40.245232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:40.245238 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:40.245298 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:40.245526 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.245916 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.246495 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.246660 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.246728 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.248385 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:40.248386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:40.248387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:40.248393 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:40.249873 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.250238 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.250239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.250239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.250303 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:40.250486 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:40.250487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:40.250487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:40.250580 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:40.250645 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:40.250792 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 22:52:40.250858 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 22:52:40.250937 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:40.245073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:40261 (local address 127.0.0.1:50692)
0808 22:52:40.245114 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:40.245123 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:40.245157 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:40.245305 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:40.245306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:40.245425 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:40.245454 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.245455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.245994 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.245995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.246421 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:40.246422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:40.246726 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:40.246727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:40.246783 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:40.247137 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:40.247148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:40.248284 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:40.249959 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.249960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.249960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.250158 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.250360 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:40.250361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:40.250361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:40.250415 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:40.250650 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:40.250651 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:40.250687 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 22:52:40.250776 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 22:52:40.250858 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 22:52:40.252638 11220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: Bootstrap starting.
I20260808 22:52:40.253202 11220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:40.253476 11220 log.cc:824] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:40.254112 11220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: No bootstrap required, opened a new log
I20260808 22:52:40.255507 11220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.256093 11220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:40.256146 11220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2734124472bc4d2aaf5aaad526dd5248, State: Initialized, Role: FOLLOWER
I20260808 22:52:40.256258 11220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.256505 11408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.256546 11220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:40.256613 11408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:52:40.258314 11424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:52:40.258433 11424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:52:40.260392 11118 master_runner.cc:263] No leader master found from master 2734124472bc4d2aaf5aaad526dd5248
I20260808 22:52:40.260445 11118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 22:52:40.261864 11148 tablet.cc:2369] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:52:40.269500 11408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:40.269567 11408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.269743 11408 leader_election.cc:288] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f792c7980b546049b7045a7a189036d (127.22.30.62:43515), 388456b360b344ea855cda3778654a10 (127.22.30.60:40261)
I20260808 22:52:40.270054 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2734124472bc4d2aaf5aaad526dd5248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388456b360b344ea855cda3778654a10" is_pre_election: true
I20260808 22:52:40.270169 11302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2734124472bc4d2aaf5aaad526dd5248 in term 0.
I20260808 22:52:40.270395 11143 leader_election.cc:302] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2734124472bc4d2aaf5aaad526dd5248, 388456b360b344ea855cda3778654a10; no voters:
I20260808 22:52:40.270542 11408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:52:40.270586 11408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:52:40.270606 11408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:40.271641 11068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2734124472bc4d2aaf5aaad526dd5248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f792c7980b546049b7045a7a189036d" is_pre_election: true
I20260808 22:52:40.271757 11068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2734124472bc4d2aaf5aaad526dd5248 in term 0.
I20260808 22:52:40.271353 11408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.272120 11408 leader_election.cc:288] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [CANDIDATE]: Term 1 election: Requested vote from peers 0f792c7980b546049b7045a7a189036d (127.22.30.62:43515), 388456b360b344ea855cda3778654a10 (127.22.30.60:40261)
I20260808 22:52:40.272334 11302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2734124472bc4d2aaf5aaad526dd5248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388456b360b344ea855cda3778654a10"
I20260808 22:52:40.272408 11302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:40.273454 11302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2734124472bc4d2aaf5aaad526dd5248 in term 1.
I20260808 22:52:40.274076 11143 leader_election.cc:302] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2734124472bc4d2aaf5aaad526dd5248, 388456b360b344ea855cda3778654a10; no voters:
I20260808 22:52:40.274457 11068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2734124472bc4d2aaf5aaad526dd5248" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f792c7980b546049b7045a7a189036d"
I20260808 22:52:40.274545 11068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:40.275312 11068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2734124472bc4d2aaf5aaad526dd5248 in term 1.
I20260808 22:52:40.275554 11408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:40.275693 11408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [term 1 LEADER]: Becoming Leader. State: Replica: 2734124472bc4d2aaf5aaad526dd5248, State: Running, Role: LEADER
I20260808 22:52:40.275779 11408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } }
I20260808 22:52:40.276070 11408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2734124472bc4d2aaf5aaad526dd5248. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.276146 11408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:40.276435 11428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:40.276625 11428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:40.278576 11302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [term 1 FOLLOWER]: Refusing update from remote peer 2734124472bc4d2aaf5aaad526dd5248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:40.278795 11068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [term 1 FOLLOWER]: Refusing update from remote peer 2734124472bc4d2aaf5aaad526dd5248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:40.279140 11408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:40.280407 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2734124472bc4d2aaf5aaad526dd5248. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.280480 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:52:40.280642 11408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:40.281697 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2734124472bc4d2aaf5aaad526dd5248. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.281785 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:52:40.283070 11408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.283139 11408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:40.283568 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.283638 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:52:40.283845 11429 mvcc.cc:205] Tried to move back new op lower bound from 7316396278898302976 to 7316396278889992192. Current Snapshot: MvccSnapshot[applied={T|T < 7316396278898302976}]
I20260808 22:52:40.283987 11430 mvcc.cc:205] Tried to move back new op lower bound from 7316396278898302976 to 7316396278889992192. Current Snapshot: MvccSnapshot[applied={T|T < 7316396278898302976}]
I20260808 22:52:40.284962 11426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.285043 11426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:40.285501 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2734124472bc4d2aaf5aaad526dd5248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f792c7980b546049b7045a7a189036d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43515 } } peers { permanent_uuid: "2734124472bc4d2aaf5aaad526dd5248" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 38485 } } peers { permanent_uuid: "388456b360b344ea855cda3778654a10" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 40261 } } }
I20260808 22:52:40.285579 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:52:40.285828 11433 mvcc.cc:205] Tried to move back new op lower bound from 7316396278898302976 to 7316396278889992192. Current Snapshot: MvccSnapshot[applied={T|T < 7316396278898302976}]
I20260808 22:52:40.287735 11428 catalog_manager.cc:1347] Generated new cluster ID: c3f15f0fe81741ad9b6a24ab9f881cf1
I20260808 22:52:40.287789 11428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:52:40.298627 11428 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:52:40.299721 11428 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:52:40.304989 11428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2734124472bc4d2aaf5aaad526dd5248: Generated new TSK 0
I20260808 22:52:40.305150 11428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:52:41.212284 11371 catalog_manager.cc:1259] Loaded cluster ID: c3f15f0fe81741ad9b6a24ab9f881cf1
I20260808 22:52:41.212340 11371 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: loading cluster ID for follower catalog manager: success
I20260808 22:52:41.213050 11371 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: acquiring CA information for follower catalog manager: success
I20260808 22:52:41.213290 11371 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 388456b360b344ea855cda3778654a10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:52:41.223148 11405 catalog_manager.cc:1259] Loaded cluster ID: c3f15f0fe81741ad9b6a24ab9f881cf1
I20260808 22:52:41.223199 11405 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: loading cluster ID for follower catalog manager: success
I20260808 22:52:41.223886 11405 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: acquiring CA information for follower catalog manager: success
I20260808 22:52:41.224094 11405 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f792c7980b546049b7045a7a189036d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:52:41.357079 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11009
I20260808 22:52:41.367470 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11118
I20260808 22:52:41.377842 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11232
2026-08-08T22:52:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2911 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:52:41 dist-test-slave-x4qh krb5kdc[11455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:52:41 dist-test-slave-x4qh krb5kdc[11455](info): set up 2 sockets
Aug 08 22:52:41 dist-test-slave-x4qh krb5kdc[11455](info): commencing operation
krb5kdc: starting...
W20260808 22:52:42.521646 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:52:42 dist-test-slave-x4qh krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:52:42.629658 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:46313
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:42.703915 11475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:42.704077 11475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:42.704095 11475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:42.705145 11475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:42.705183 11475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:42.705214 11475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:42.705251 11475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:42.705281 11475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:42.706583 11475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:46313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:42.706794 11475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:42.707022 11475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:42.711570 11483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:42.711580 11486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:42.711599 11481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:42.711612 11475 server_base.cc:1034] running on GCE node
I20260808 22:52:42.712054 11475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:42.712308 11475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:42.713462 11475 hybrid_clock.cc:648] HybridClock initialized: now 1786229562713509 us; error 85 us; skew 500 ppm
Aug 08 22:52:42 dist-test-slave-x4qh krb5kdc[11455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229562, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:42.715767 11475 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:42.716138 11475 webserver.cc:460] Webserver started at http://127.22.30.62:36621/ using document root <none> and password file <none>
I20260808 22:52:42.716337 11475 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:42.716399 11475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:42.716533 11475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:42.717378 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "5e6a901daaf44f27a57e5e9f968124a4"
format_stamp: "Formatted at 2026-08-08 22:52:42 on dist-test-slave-x4qh"
server_key: "dadbea0c378af22550b59dfb7ed13083"
server_key_iv: "9f4a5fe2a1f95e85896c184c07833359"
server_key_version: "encryptionkey@0"
I20260808 22:52:42.717684 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "5e6a901daaf44f27a57e5e9f968124a4"
format_stamp: "Formatted at 2026-08-08 22:52:42 on dist-test-slave-x4qh"
server_key: "dadbea0c378af22550b59dfb7ed13083"
server_key_iv: "9f4a5fe2a1f95e85896c184c07833359"
server_key_version: "encryptionkey@0"
I20260808 22:52:42.718864 11475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:52:42.719563 11494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:42.719722 11475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:42.719834 11475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "5e6a901daaf44f27a57e5e9f968124a4"
format_stamp: "Formatted at 2026-08-08 22:52:42 on dist-test-slave-x4qh"
server_key: "dadbea0c378af22550b59dfb7ed13083"
server_key_iv: "9f4a5fe2a1f95e85896c184c07833359"
server_key_version: "encryptionkey@0"
I20260808 22:52:42.719913 11475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:42.727643 11475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:42.729679 11475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:42.729856 11475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:42.733590 11475 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:46313
I20260808 22:52:42.733637 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:46313 every 8 connection(s)
I20260808 22:52:42.733964 11475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:42.734724 11575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:42.736903 11575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4: Bootstrap starting.
I20260808 22:52:42.737450 11575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:42.737742 11575 log.cc:824] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:42.738392 11575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4: No bootstrap required, opened a new log
I20260808 22:52:42.739916 11575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6a901daaf44f27a57e5e9f968124a4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46313 } }
I20260808 22:52:42.740044 11575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:42.740087 11575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6a901daaf44f27a57e5e9f968124a4, State: Initialized, Role: FOLLOWER
I20260808 22:52:42.740235 11575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6a901daaf44f27a57e5e9f968124a4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46313 } }
I20260808 22:52:42.740314 11575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:52:42.740366 11575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:52:42.740413 11575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:42.740917 11575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6a901daaf44f27a57e5e9f968124a4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46313 } }
I20260808 22:52:42.741026 11575 leader_election.cc:302] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e6a901daaf44f27a57e5e9f968124a4; no voters:
I20260808 22:52:42.741209 11575 leader_election.cc:288] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:52:42.741236 11580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:42.741420 11580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [term 1 LEADER]: Becoming Leader. State: Replica: 5e6a901daaf44f27a57e5e9f968124a4, State: Running, Role: LEADER
I20260808 22:52:42.741441 11575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:42.741545 11580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6a901daaf44f27a57e5e9f968124a4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46313 } }
I20260808 22:52:42.741942 11583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e6a901daaf44f27a57e5e9f968124a4. Latest consensus state: current_term: 1 leader_uuid: "5e6a901daaf44f27a57e5e9f968124a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6a901daaf44f27a57e5e9f968124a4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46313 } } }
I20260808 22:52:42.742066 11582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e6a901daaf44f27a57e5e9f968124a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e6a901daaf44f27a57e5e9f968124a4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 46313 } } }
I20260808 22:52:42.742082 11583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:42.742158 11582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:42.742518 11591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:42.742918 11591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:42.743829 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11475
I20260808 22:52:42.743945 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:52:42.744151 22648 external_mini_cluster.cc:1428] Setting key f0f1c0261da0d80f7a9fb7d154fb1aa9
I20260808 22:52:42.745010 11591 catalog_manager.cc:1347] Generated new cluster ID: 47eaea98102845679e54b5224ea8161c
I20260808 22:52:42.745070 11591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:52:42 dist-test-slave-x4qh krb5kdc[11455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:42.755172 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:42.744574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.22.30.62:46313)
0808 22:52:42.744726 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:42.744734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:42.745183 (+ 449us) server_negotiation.cc:407] Connection header received
0808 22:52:42.745329 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:42.745337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:42.745389 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:42.745535 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:42.746203 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:42.747543 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:42.748565 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:42.749180 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:42.749302 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:42.751893 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:42.751896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:42.751897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:42.751907 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:42.753748 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:42.754111 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:42.754113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:42.754113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:42.754204 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:42.754406 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:42.754407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:42.754407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:42.754638 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:42.754756 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:42.754868 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 22:52:42.754973 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260808 22:52:42.756026 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:52:42.766958 11591 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:52:42.767495 11591 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:52:42.770185 11591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e6a901daaf44f27a57e5e9f968124a4: Generated new TSK 0
I20260808 22:52:42.770354 11591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:52:42.809868 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11475
2026-08-08T22:52:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1433 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:52:42 dist-test-slave-x4qh krb5kdc[11622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:52:42 dist-test-slave-x4qh krb5kdc[11622](info): set up 2 sockets
Aug 08 22:52:42 dist-test-slave-x4qh krb5kdc[11622](info): commencing operation
krb5kdc: starting...
W20260808 22:52:43.987635 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:52:44.098157 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34085
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:44.172317 11639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:44.172549 11639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:44.172598 11639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:44.173728 11639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:44.173789 11639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:44.173813 11639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:44.173839 11639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:44.173861 11639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:44.175256 11639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:44.175526 11639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:44.175750 11639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:44.180245 11644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.180264 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.180264 11647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:44.180441 11639 server_base.cc:1034] running on GCE node
I20260808 22:52:44.180693 11639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:44.181006 11639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:44.182148 11639 hybrid_clock.cc:648] HybridClock initialized: now 1786229564182130 us; error 33 us; skew 500 ppm
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:44.184557 11639 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:44.184975 11639 webserver.cc:460] Webserver started at http://127.22.30.62:41649/ using document root <none> and password file <none>
I20260808 22:52:44.185165 11639 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:44.185205 11639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:44.185286 11639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:44.186239 11639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "d162b904e57b49c99f7986fd7b60b593"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "090e3725e903397e2efc730ac2bf213b"
server_key_iv: "1607f5ce38744097b55bdbcb4be9c565"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.186507 11639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "d162b904e57b49c99f7986fd7b60b593"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "090e3725e903397e2efc730ac2bf213b"
server_key_iv: "1607f5ce38744097b55bdbcb4be9c565"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.187642 11639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:52:44.188256 11660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.188423 11639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:44.188503 11639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d162b904e57b49c99f7986fd7b60b593"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "090e3725e903397e2efc730ac2bf213b"
server_key_iv: "1607f5ce38744097b55bdbcb4be9c565"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.188546 11639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:44.207849 11639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:44.209438 11639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:44.209538 11639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:44.213186 11639 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:34085
I20260808 22:52:44.213240 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:34085 every 8 connection(s)
I20260808 22:52:44.213567 11639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:44.214498 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:44.216562 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap starting.
I20260808 22:52:44.217111 11746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:44.217396 11746 log.cc:824] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:44.218086 11746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: No bootstrap required, opened a new log
I20260808 22:52:44.219686 11746 raft_consensus.cc:348] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:44.219832 11746 raft_consensus.cc:374] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:44.219878 11746 raft_consensus.cc:729] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d162b904e57b49c99f7986fd7b60b593, State: Initialized, Role: FOLLOWER
I20260808 22:52:44.220062 11746 consensus_queue.cc:260] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:44.220163 11746 raft_consensus.cc:388] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:52:44.220218 11746 raft_consensus.cc:482] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:52:44.220292 11746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:44.221050 11746 raft_consensus.cc:504] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:44.221164 11746 leader_election.cc:302] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d162b904e57b49c99f7986fd7b60b593; no voters:
I20260808 22:52:44.221345 11746 leader_election.cc:288] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:52:44.221372 11749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:44.221521 11749 raft_consensus.cc:686] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 LEADER]: Becoming Leader. State: Replica: d162b904e57b49c99f7986fd7b60b593, State: Running, Role: LEADER
I20260808 22:52:44.221580 11746 sys_catalog.cc:564] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:44.221653 11749 consensus_queue.cc:237] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:44.221997 11750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d162b904e57b49c99f7986fd7b60b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } } }
I20260808 22:52:44.222093 11750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:44.222004 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d162b904e57b49c99f7986fd7b60b593. Latest consensus state: current_term: 1 leader_uuid: "d162b904e57b49c99f7986fd7b60b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } } }
I20260808 22:52:44.222227 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:44.222383 11764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:44.222901 11764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:44.223004 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11639
I20260808 22:52:44.223129 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:52:44.223331 22648 external_mini_cluster.cc:1428] Setting key 23241d0fc329135404d65920e8950b11
I20260808 22:52:44.224448 11764 catalog_manager.cc:1347] Generated new cluster ID: 9038df9f33d44d55b34e7146096805fd
I20260808 22:52:44.224503 11764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:44.235958 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.223726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.22.30.62:34085)
0808 22:52:44.224261 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.224268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.224305 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:52:44.227439 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.227445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.227490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.227618 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:44.228224 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.229162 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.229915 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.230307 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.230419 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.232685 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:44.232693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:44.232694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:44.232703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:44.234401 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.234755 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.234757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.234758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.234843 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.235065 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.235066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.235066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.235284 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:44.235460 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.235569 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.235792 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":75,"threads_started":1}
I20260808 22:52:44.254175 11764 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:52:44.254773 11764 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:52:44.259817 11764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Generated new TSK 0
I20260808 22:52:44.259963 11764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:52:44.287966 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:34085
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:44.365604 11786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:44.365826 11786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:44.365865 11786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:44.366978 11786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:44.367039 11786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:44.367079 11786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:52:44.368554 11786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:44.368834 11786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:44.369061 11786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:44.373876 11794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.373883 11793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.373886 11801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:44.374168 11786 server_base.cc:1034] running on GCE node
I20260808 22:52:44.374369 11786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:44.374601 11786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:44.375752 11786 hybrid_clock.cc:648] HybridClock initialized: now 1786229564375742 us; error 39 us; skew 500 ppm
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:44.378193 11786 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:52:44.378621 11786 webserver.cc:460] Webserver started at http://127.22.30.1:41607/ using document root <none> and password file <none>
I20260808 22:52:44.378815 11786 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:44.378855 11786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:44.378937 11786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:44.379884 11786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "1cf759a6977341b9a9ad0a216f045c24"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "614c22c50f28930de5ec2de2735660d5"
server_key_iv: "9236865903f00955f0cc359bae1a3c95"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.380151 11786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "1cf759a6977341b9a9ad0a216f045c24"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "614c22c50f28930de5ec2de2735660d5"
server_key_iv: "9236865903f00955f0cc359bae1a3c95"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.381249 11786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:52:44.381865 11810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.382071 11786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:44.382151 11786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "1cf759a6977341b9a9ad0a216f045c24"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "614c22c50f28930de5ec2de2735660d5"
server_key_iv: "9236865903f00955f0cc359bae1a3c95"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.382203 11786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:44.398948 11786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:44.400619 11786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:44.400717 11786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:44.400877 11786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:44.401083 11786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:44.401109 11786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.401127 11786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:44.401139 11786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.406657 11786 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:36837
I20260808 22:52:44.406761 11973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:36837 every 8 connection(s)
I20260808 22:52:44.407029 11786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:44.412743 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11786
I20260808 22:52:44.412834 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:52:44.412997 22648 external_mini_cluster.cc:1428] Setting key 4b6608ef2502b927cfc607c8597c4aff
I20260808 22:52:44.415741 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.407725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60321 (local address 127.22.30.62:34085)
0808 22:52:44.407768 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.407770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.407978 (+ 208us) server_negotiation.cc:407] Connection header received
0808 22:52:44.408429 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.408430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.408445 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.408501 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:44.409376 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.409866 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.410789 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.410979 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.411053 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.412919 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:44.412920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:44.412921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:44.412927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:44.414336 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.414793 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.414794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.414794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.414839 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.415105 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.415106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.415106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.415188 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:44.415232 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.415538 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.415633 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:52:44.415823 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.407699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:60321)
0808 22:52:44.407860 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.407910 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.408331 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.408561 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.408564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.409098 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:44.409263 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.409267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.409971 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.409972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.410693 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.410695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.411074 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.411075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.411199 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.411614 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:44.411621 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:44.412822 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:44.414414 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:44.414421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:44.414423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:44.414711 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:44.414895 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:44.414896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:44.414897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:44.415036 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:44.415244 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:44.415249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:44.415399 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.415498 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.415656 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260808 22:52:44.416178 11975 heartbeater.cc:347] Connected to a master server at 127.22.30.62:34085
I20260808 22:52:44.416258 11975 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:44.416399 11975 heartbeater.cc:511] Master 127.22.30.62:34085 requested a full tablet report, sending...
I20260808 22:52:44.416888 11687 ts_manager.cc:194] Registered new tserver with Master: 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837)
I20260808 22:52:44.417433 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:60321
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:52:44.436180 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:34085
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:44.509382 11982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:44.509573 11982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:44.509609 11982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:44.510613 11982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:44.510649 11982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:44.510672 11982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:52:44.512027 11982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:44.512187 11982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:44.512346 11982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:44.517038 11988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.517081 11987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:44.517133 11982 server_base.cc:1034] running on GCE node
W20260808 22:52:44.517059 11993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:44.517465 11982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:44.517694 11982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:44.518855 11982 hybrid_clock.cc:648] HybridClock initialized: now 1786229564518852 us; error 35 us; skew 500 ppm
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:44.521189 11982 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:52:44.521622 11982 webserver.cc:460] Webserver started at http://127.22.30.2:39091/ using document root <none> and password file <none>
I20260808 22:52:44.521828 11982 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:44.521890 11982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:44.522014 11982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:44.523068 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "aa13000474a04c848167062a50b80d38"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "b29b38bdea63c526f773b802586895d4"
server_key_iv: "efa9fd2f52f85e223d156a82f08eea4c"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.523510 11982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "aa13000474a04c848167062a50b80d38"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "b29b38bdea63c526f773b802586895d4"
server_key_iv: "efa9fd2f52f85e223d156a82f08eea4c"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.524786 11982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:44.525489 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.525661 11982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:44.525750 11982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "aa13000474a04c848167062a50b80d38"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "b29b38bdea63c526f773b802586895d4"
server_key_iv: "efa9fd2f52f85e223d156a82f08eea4c"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.525830 11982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:44.548333 11982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:44.550132 11982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:44.550280 11982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:44.550477 11982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:44.550781 11982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:44.550815 11982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.550836 11982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:44.550892 11982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.556313 11982 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:45619
I20260808 22:52:44.556388 12193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:45619 every 8 connection(s)
I20260808 22:52:44.556668 11982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:52:44.560343 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 11982
I20260808 22:52:44.560463 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:52:44.560658 22648 external_mini_cluster.cc:1428] Setting key 98b11297c049ef0cdd5992287242bffe
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:44.565743 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.557350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:43353)
0808 22:52:44.557524 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.557556 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.557980 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.558167 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.558170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.558657 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:44.558850 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.558854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.559528 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.559529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.560224 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.560225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.560708 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.560709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.560837 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.561206 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:44.561213 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:44.562867 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:44.564475 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:44.564481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:44.564483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:44.564707 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:44.564916 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:44.564916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:44.564916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:44.565031 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:44.565307 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:44.565311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:44.565414 (+ 103us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.565492 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.565599 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260808 22:52:44.565761 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.557361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:43353 (local address 127.22.30.62:34085)
0808 22:52:44.557426 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.557428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.557630 (+ 202us) server_negotiation.cc:407] Connection header received
0808 22:52:44.558054 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.558055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.558064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.558114 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:44.558918 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.559454 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.560318 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.560612 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.560680 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.562961 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:44.562963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:44.562963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:44.562969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:44.564382 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.564785 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.564786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.564787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.564850 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.565092 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.565093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.565093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.565243 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:44.565312 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.565534 (+ 222us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.565624 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:52:44.566329 12195 heartbeater.cc:347] Connected to a master server at 127.22.30.62:34085
I20260808 22:52:44.566439 12195 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:44.566613 12195 heartbeater.cc:511] Master 127.22.30.62:34085 requested a full tablet report, sending...
I20260808 22:52:44.566938 11687 ts_manager.cc:194] Registered new tserver with Master: aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
I20260808 22:52:44.567260 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:43353
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:52:44.585801 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:34085
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:44.662201 12204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:44.662384 12204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:44.662420 12204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:44.663439 12204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:44.663489 12204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:44.663527 12204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:52:44.664887 12204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:44.665126 12204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:44.665320 12204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:44.669651 12213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.669724 12212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:44.669662 12216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:44.669915 12204 server_base.cc:1034] running on GCE node
I20260808 22:52:44.670095 12204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:44.670305 12204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:44.671433 12204 hybrid_clock.cc:648] HybridClock initialized: now 1786229564671418 us; error 27 us; skew 500 ppm
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:44.673646 12204 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:52:44.674024 12204 webserver.cc:460] Webserver started at http://127.22.30.3:39349/ using document root <none> and password file <none>
I20260808 22:52:44.674237 12204 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:44.674305 12204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:44.674423 12204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:52:44.675302 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "34c714028e71431b9f7342f66734a676"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "2c2cb8712c58dc24b8fd8de01a190119"
server_key_iv: "0c6390b857ab407bc1f9eb48a1475629"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.675702 12204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "34c714028e71431b9f7342f66734a676"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "2c2cb8712c58dc24b8fd8de01a190119"
server_key_iv: "0c6390b857ab407bc1f9eb48a1475629"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.676905 12204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:44.677515 12224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.677676 12204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:44.677760 12204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "34c714028e71431b9f7342f66734a676"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "2c2cb8712c58dc24b8fd8de01a190119"
server_key_iv: "0c6390b857ab407bc1f9eb48a1475629"
server_key_version: "encryptionkey@0"
I20260808 22:52:44.677842 12204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:44.686481 12204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:44.688158 12204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:44.688311 12204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:44.688467 12204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:44.688731 12204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:44.688802 12204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.688859 12204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:44.688887 12204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:44.694510 12204 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:37631
I20260808 22:52:44.694597 12395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:37631 every 8 connection(s)
I20260808 22:52:44.694851 12204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:52:44 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229564, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:44.700520 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 12204
I20260808 22:52:44.700636 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:52:44.700830 22648 external_mini_cluster.cc:1428] Setting key 0606925b0672f60e92d7a7ca30332b33
I20260808 22:52:44.703440 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.695616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:37863 (local address 127.22.30.62:34085)
0808 22:52:44.695658 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.695660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.695823 (+ 163us) server_negotiation.cc:407] Connection header received
0808 22:52:44.696222 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.696223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.696231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.696285 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:44.697110 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.697592 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.698495 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.698666 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.698717 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.700801 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:44.700802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:44.700803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:44.700809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:44.702377 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.702698 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.702699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.702699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.702761 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.702952 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.702952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.702953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.703056 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:44.703100 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.703280 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.703334 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:52:44.703581 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.695567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:37863)
0808 22:52:44.695711 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.695747 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.696147 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.696340 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.696343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.696861 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:44.697029 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.697033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.697672 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.697672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.698423 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.698425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.698735 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.698736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.698858 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.699239 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:44.699247 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:44.700701 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:44.702455 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:44.702461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:44.702462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:44.702641 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:44.702811 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:44.702811 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:44.702812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:44.702887 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:44.703092 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:44.703094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:44.703182 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.703248 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.703347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260808 22:52:44.703974 12396 heartbeater.cc:347] Connected to a master server at 127.22.30.62:34085
I20260808 22:52:44.704062 12396 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:44.704211 12396 heartbeater.cc:511] Master 127.22.30.62:34085 requested a full tablet report, sending...
I20260808 22:52:44.704572 11687 ts_manager.cc:194] Registered new tserver with Master: 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631)
I20260808 22:52:44.704929 11687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:37863
I20260808 22:52:44.712697 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:52:44.719121 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.714070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.22.30.62:34085)
0808 22:52:44.714141 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.714143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.714220 (+ 77us) server_negotiation.cc:407] Connection header received
0808 22:52:44.714278 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.714279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.714287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.714322 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:44.714790 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.715214 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.715909 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.716068 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.716115 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.717066 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:44.717067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:44.717067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:44.717072 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:44.718216 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.718498 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.718498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.718499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.718545 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:44.718692 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:44.718692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:44.718692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:44.718835 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:44.718902 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.718989 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.719044 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:52:44.720697 11687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:52:44.722272 11687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:52:44.728068 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.724327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:37631 (local address 127.0.0.1:46226)
0808 22:52:44.724682 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.724699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.724741 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.725142 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.725143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.725145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:44.725172 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.725173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.726318 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.726319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.726982 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.726984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.727763 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.727841 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.727859 (+ 18us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.727953 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.728004 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":62,"threads_started":1}
I20260808 22:52:44.728101 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.724510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.22.30.3:37631)
0808 22:52:44.724998 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.725003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.725012 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:52:44.725043 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.725045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.725058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.725131 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:44.725250 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.726230 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727064 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.727646 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727741 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.727804 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.727968 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.728033 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":87,"threads_started":1}
I20260808 22:52:44.728276 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.724300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36837 (local address 127.0.0.1:45086)
0808 22:52:44.724638 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.724646 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.724681 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.724887 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.724890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.724891 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:44.724950 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.724952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.725975 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.725976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.727148 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.727151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.728032 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.728033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.728105 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.728107 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.728187 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.728218 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":30,"spinlock_wait_cycles":384,"thread_start_us":60,"threads_started":1}
I20260808 22:52:44.728425 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.724525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.22.30.2:45619)
0808 22:52:44.724831 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.724834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.724841 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:52:44.724871 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.724872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.724882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.724963 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:44.725071 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.726120 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727236 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.727886 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727950 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.727996 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.728312 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.728351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":80,"threads_started":1}
I20260808 22:52:44.728463 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.724518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.22.30.1:36837)
0808 22:52:44.724708 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.724713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.724722 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:52:44.724763 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.724765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.724793 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.724869 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:44.725031 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.725882 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727236 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.727885 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.727950 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.727996 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.728342 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.728388 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260808 22:52:44.728276 12409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.724300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:45619 (local address 127.0.0.1:48638)
0808 22:52:44.724453 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.724487 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.724544 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.724962 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.724963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.724965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:44.725018 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.725020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.726217 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.726218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.727151 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.727152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.728031 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.728032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.728097 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.728101 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.728149 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.728193 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260808 22:52:44.729085 12291 tablet_service.cc:1467] Processing CreateTablet for tablet 2041c3a8071a45dbb03c959c029a737e (DEFAULT_TABLE table=test-table [id=1ee92031ead7412d921f42ea44391842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:44.729182 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 2041c3a8071a45dbb03c959c029a737e (DEFAULT_TABLE table=test-table [id=1ee92031ead7412d921f42ea44391842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:44.729183 12081 tablet_service.cc:1467] Processing CreateTablet for tablet 2041c3a8071a45dbb03c959c029a737e (DEFAULT_TABLE table=test-table [id=1ee92031ead7412d921f42ea44391842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:44.729439 12291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2041c3a8071a45dbb03c959c029a737e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:44.729492 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2041c3a8071a45dbb03c959c029a737e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:44.729492 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2041c3a8071a45dbb03c959c029a737e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:44.732506 12416 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap starting.
I20260808 22:52:44.732677 12415 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Bootstrap starting.
I20260808 22:52:44.732698 12417 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Bootstrap starting.
I20260808 22:52:44.733238 12417 tablet_bootstrap.cc:654] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:44.733242 12415 tablet_bootstrap.cc:654] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:44.733749 12415 log.cc:824] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:44.733719 12417 log.cc:824] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:44.734489 12417 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: No bootstrap required, opened a new log
I20260808 22:52:44.734496 12415 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: No bootstrap required, opened a new log
I20260808 22:52:44.734556 12415 ts_tablet_manager.cc:1397] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:52:44.734608 12416 tablet_bootstrap.cc:654] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:44.734614 12417 ts_tablet_manager.cc:1397] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:52:44.735118 12416 log.cc:824] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:44.736470 12417 raft_consensus.cc:348] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.736629 12417 raft_consensus.cc:374] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:44.736698 12417 raft_consensus.cc:729] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa13000474a04c848167062a50b80d38, State: Initialized, Role: FOLLOWER
I20260808 22:52:44.736644 12415 raft_consensus.cc:348] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.736793 12415 raft_consensus.cc:374] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:44.736826 12416 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: No bootstrap required, opened a new log
I20260808 22:52:44.736840 12415 raft_consensus.cc:729] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c714028e71431b9f7342f66734a676, State: Initialized, Role: FOLLOWER
I20260808 22:52:44.736842 12417 consensus_queue.cc:260] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.736923 12416 ts_tablet_manager.cc:1397] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 22:52:44.736935 12415 consensus_queue.cc:260] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.737188 12396 heartbeater.cc:503] Master 127.22.30.62:34085 was elected leader, sending a full tablet report...
I20260808 22:52:44.737182 12417 ts_tablet_manager.cc:1428] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 22:52:44.737287 12195 heartbeater.cc:503] Master 127.22.30.62:34085 was elected leader, sending a full tablet report...
I20260808 22:52:44.737419 12415 ts_tablet_manager.cc:1428] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 22:52:44.738679 12416 raft_consensus.cc:348] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.738816 12416 raft_consensus.cc:374] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:44.738837 12416 raft_consensus.cc:729] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf759a6977341b9a9ad0a216f045c24, State: Initialized, Role: FOLLOWER
I20260808 22:52:44.738965 12416 consensus_queue.cc:260] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.739239 12416 ts_tablet_manager.cc:1428] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:52:44.739244 11975 heartbeater.cc:503] Master 127.22.30.62:34085 was elected leader, sending a full tablet report...
W20260808 22:52:44.807250 12196 tablet.cc:2369] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:52:44.847362 12425 raft_consensus.cc:482] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:44.847463 12425 raft_consensus.cc:504] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.847816 12425 leader_election.cc:288] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837), 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631)
I20260808 22:52:44.850416 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.848026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:55951 (local address 127.22.30.1:36837)
0808 22:52:44.848068 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.848070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.848105 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:52:44.848134 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.848135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.848146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.848222 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:44.848387 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.848841 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.849721 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.850088 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.850169 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.850179 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.850322 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.850358 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:52:44.850417 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.847945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36837 (local address 127.22.30.2:55951)
0808 22:52:44.848008 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.848019 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.848064 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.848229 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.848230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.848231 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:44.848306 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.848307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.848924 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.848926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.849629 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.849630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.850169 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.850170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.850252 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.850254 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.850314 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.850356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 22:52:44.850581 12428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.848020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:37631 (local address 127.22.30.2:50559)
0808 22:52:44.848161 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.848169 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:44.848240 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:44.848411 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:44.848412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:44.848413 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:44.848479 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.848480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.849091 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.849092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.849804 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:44.849806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.850338 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.850339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:44.850410 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.850414 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:52:44.850473 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 22:52:44.850510 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":8,"thread_start_us":99,"threads_started":1}
I20260808 22:52:44.850569 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.848083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:50559 (local address 127.22.30.3:37631)
0808 22:52:44.848124 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.848126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.848243 (+ 117us) server_negotiation.cc:407] Connection header received
0808 22:52:44.848319 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.848320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.848331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.848409 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:44.848553 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.849005 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.849895 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.850248 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.850325 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.850334 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.850480 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.850512 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:52:44.850749 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2041c3a8071a45dbb03c959c029a737e" candidate_uuid: "aa13000474a04c848167062a50b80d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf759a6977341b9a9ad0a216f045c24" is_pre_election: true
I20260808 22:52:44.850903 11904 raft_consensus.cc:2454] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa13000474a04c848167062a50b80d38 in term 0.
I20260808 22:52:44.851050 12323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2041c3a8071a45dbb03c959c029a737e" candidate_uuid: "aa13000474a04c848167062a50b80d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c714028e71431b9f7342f66734a676" is_pre_election: true
I20260808 22:52:44.851128 12008 leader_election.cc:302] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24, aa13000474a04c848167062a50b80d38; no voters:
I20260808 22:52:44.851217 12323 raft_consensus.cc:2454] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa13000474a04c848167062a50b80d38 in term 0.
I20260808 22:52:44.851254 12425 raft_consensus.cc:2781] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:52:44.851305 12425 raft_consensus.cc:482] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:52:44.851329 12425 raft_consensus.cc:3037] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:44.852030 12425 raft_consensus.cc:504] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.852162 12425 leader_election.cc:288] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [CANDIDATE]: Term 1 election: Requested vote from peers 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837), 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631)
I20260808 22:52:44.852284 12323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2041c3a8071a45dbb03c959c029a737e" candidate_uuid: "aa13000474a04c848167062a50b80d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c714028e71431b9f7342f66734a676"
I20260808 22:52:44.852279 11904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2041c3a8071a45dbb03c959c029a737e" candidate_uuid: "aa13000474a04c848167062a50b80d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf759a6977341b9a9ad0a216f045c24"
I20260808 22:52:44.852347 12323 raft_consensus.cc:3037] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:44.852347 11904 raft_consensus.cc:3037] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:44.853111 12323 raft_consensus.cc:2454] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa13000474a04c848167062a50b80d38 in term 1.
I20260808 22:52:44.853111 11904 raft_consensus.cc:2454] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa13000474a04c848167062a50b80d38 in term 1.
I20260808 22:52:44.853266 12008 leader_election.cc:302] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24, aa13000474a04c848167062a50b80d38; no voters:
I20260808 22:52:44.853391 12425 raft_consensus.cc:2781] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:44.853519 12425 raft_consensus.cc:686] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 1 LEADER]: Becoming Leader. State: Replica: aa13000474a04c848167062a50b80d38, State: Running, Role: LEADER
I20260808 22:52:44.853609 12425 consensus_queue.cc:237] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:44.854272 11687 catalog_manager.cc:5495] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa13000474a04c848167062a50b80d38 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "aa13000474a04c848167062a50b80d38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } health_report { overall_health: UNKNOWN } } }
I20260808 22:52:44.873162 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.871000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.22.30.2:45619)
0808 22:52:44.871039 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:44.871041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:44.871049 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:52:44.871110 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:44.871111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:44.871131 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:44.871196 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:52:44.871326 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.871732 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.872461 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:44.872632 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:44.872697 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:44.872861 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:52:44.872947 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:52:44.873025 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:44.873061 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 22:52:44.873078 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:52:44.874552 12323 raft_consensus.cc:1261] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Refusing update from remote peer aa13000474a04c848167062a50b80d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:44.874552 11904 raft_consensus.cc:1261] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Refusing update from remote peer aa13000474a04c848167062a50b80d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:44.874826 12432 consensus_queue.cc:1035] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:44.874938 12425 consensus_queue.cc:1035] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:44.877473 12437 mvcc.cc:205] Tried to move back new op lower bound from 7316396297724252160 to 7316396297640812544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396297724252160}]
I20260808 22:52:44.877873 12438 mvcc.cc:205] Tried to move back new op lower bound from 7316396297724252160 to 7316396297640812544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396297724252160 or (T in {7316396297724252160})}]
I20260808 22:52:44.879755 11685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:52:44.879841 11685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:52:44.880496 11685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Sending DeleteTablet for 3 replicas of tablet 2041c3a8071a45dbb03c959c029a737e
I20260808 22:52:44.880837 12081 tablet_service.cc:1514] Processing DeleteTablet for tablet 2041c3a8071a45dbb03c959c029a737e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:44 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:48638
I20260808 22:52:44.880842 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2041c3a8071a45dbb03c959c029a737e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:44 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:45086
I20260808 22:52:44.881086 12291 tablet_service.cc:1514] Processing DeleteTablet for tablet 2041c3a8071a45dbb03c959c029a737e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:44 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46226
I20260808 22:52:44.881139 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11786
I20260808 22:52:44.881145 12450 tablet_replica.cc:330] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: stopping tablet replica
I20260808 22:52:44.881244 12450 raft_consensus.cc:2229] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:52:44.881335 12450 raft_consensus.cc:2258] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:44.884542 12452 tablet_replica.cc:330] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: stopping tablet replica
I20260808 22:52:44.884681 12452 raft_consensus.cc:2229] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:52:44.884845 12452 raft_consensus.cc:2258] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:44.886279 12452 ts_tablet_manager.cc:1905] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:44.888481 12452 ts_tablet_manager.cc:1918] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:52:44.888577 12452 log.cc:1196] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/2041c3a8071a45dbb03c959c029a737e
I20260808 22:52:44.888939 12452 ts_tablet_manager.cc:1939] T 2041c3a8071a45dbb03c959c029a737e P aa13000474a04c848167062a50b80d38: Deleting consensus metadata
I20260808 22:52:44.889416 11661 catalog_manager.cc:4841] TS aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): tablet 2041c3a8071a45dbb03c959c029a737e (table test-table [id=1ee92031ead7412d921f42ea44391842]) successfully deleted
I20260808 22:52:44.894675 12453 tablet_replica.cc:330] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: stopping tablet replica
I20260808 22:52:44.894800 12453 raft_consensus.cc:2229] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:52:44.895124 12453 raft_consensus.cc:2258] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:44.895731 12453 ts_tablet_manager.cc:1905] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:44.897007 12453 ts_tablet_manager.cc:1918] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:52:44.897090 12453 log.cc:1196] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/2041c3a8071a45dbb03c959c029a737e
I20260808 22:52:44.897336 12453 ts_tablet_manager.cc:1939] T 2041c3a8071a45dbb03c959c029a737e P 34c714028e71431b9f7342f66734a676: Deleting consensus metadata
I20260808 22:52:44.897684 11666 catalog_manager.cc:4841] TS 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): tablet 2041c3a8071a45dbb03c959c029a737e (table test-table [id=1ee92031ead7412d921f42ea44391842]) successfully deleted
W20260808 22:52:44.900650 11662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:36837 (error 108)
I20260808 22:52:44.900836 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11982
I20260808 22:52:44.901039 12409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:44.900938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36837 (local address 127.0.0.1:45100)
0808 22:52:44.900980 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:44.901007 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 22:52:44.902143 11662 catalog_manager.cc:4584] TS 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2041c3a8071a45dbb03c959c029a737e: Network error: Client connection negotiation failed: client connection to 127.22.30.1:36837: connect: Connection refused (error 111)
I20260808 22:52:44.909132 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 12204
I20260808 22:52:44.919040 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 11639
I20260808 22:52:44.928937 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34085
--webserver_interface=127.22.30.62
--webserver_port=41649
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:45.007000 12455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.007169 12455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.007189 12455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.008260 12455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:45.008301 12455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.008314 12455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.008352 12455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:45.008381 12455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:45.009732 12455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.009982 12455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.010195 12455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.018353 12464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.018384 12461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.018456 12455 server_base.cc:1034] running on GCE node
W20260808 22:52:45.018404 12469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.018826 12455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.019112 12455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:45.020244 12455 hybrid_clock.cc:648] HybridClock initialized: now 1786229565020233 us; error 27 us; skew 500 ppm
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.022540 12455 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:45.022917 12455 webserver.cc:460] Webserver started at http://127.22.30.62:41649/ using document root <none> and password file <none>
I20260808 22:52:45.023124 12455 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.023200 12455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.024511 12455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.025120 12478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.025278 12455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:45.025377 12455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d162b904e57b49c99f7986fd7b60b593"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "090e3725e903397e2efc730ac2bf213b"
server_key_iv: "1607f5ce38744097b55bdbcb4be9c565"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.025694 12455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.033658 12455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.035760 12455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.035910 12455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.039525 12455 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:34085
I20260808 22:52:45.039572 12563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:34085 every 8 connection(s)
I20260808 22:52:45.039944 12455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:45.041298 12564 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:52:45.041867 12564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap starting.
I20260808 22:52:45.044068 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 12455
I20260808 22:52:45.044412 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:36837
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:34085
--webserver_port=41607
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:45.051313 12564 log.cc:824] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:45.053095 12564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:52:45.053370 12564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap complete.
I20260808 22:52:45.055541 12564 raft_consensus.cc:348] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.055747 12564 raft_consensus.cc:729] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d162b904e57b49c99f7986fd7b60b593, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.055902 12564 consensus_queue.cc:260] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.055971 12564 raft_consensus.cc:388] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:52:45.056021 12564 raft_consensus.cc:482] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:52:45.056066 12564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:52:45.056890 12564 raft_consensus.cc:504] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.056995 12564 leader_election.cc:302] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d162b904e57b49c99f7986fd7b60b593; no voters:
I20260808 22:52:45.057194 12564 leader_election.cc:288] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:52:45.057229 12570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:52:45.057332 12570 raft_consensus.cc:686] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 LEADER]: Becoming Leader. State: Replica: d162b904e57b49c99f7986fd7b60b593, State: Running, Role: LEADER
I20260808 22:52:45.057457 12564 sys_catalog.cc:564] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:45.057485 12570 consensus_queue.cc:237] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.057857 12574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d162b904e57b49c99f7986fd7b60b593. Latest consensus state: current_term: 2 leader_uuid: "d162b904e57b49c99f7986fd7b60b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } } }
I20260808 22:52:45.057940 12574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:45.058233 12572 sys_catalog.cc:455] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d162b904e57b49c99f7986fd7b60b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } } }
I20260808 22:52:45.058346 12572 sys_catalog.cc:458] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:45.058511 12580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:45.059247 12580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:45.059417 12580 catalog_manager.cc:1259] Loaded cluster ID: 9038df9f33d44d55b34e7146096805fd
I20260808 22:52:45.059615 12580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:52:45.059849 12580 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 22:52:45.064327 12580 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 22:52:45.066556 12580 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:52:45.066749 12580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Loaded TSK: 0
I20260808 22:52:45.066982 12580 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:52:45.121884 12567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.122064 12567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.122097 12567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.123178 12567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.123239 12567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.123282 12567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:52:45.124763 12567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:36837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=41607
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.125003 12567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.125214 12567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.129930 12602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.129944 12606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.130070 12603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.130254 12567 server_base.cc:1034] running on GCE node
I20260808 22:52:45.130429 12567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.130684 12567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:45.131840 12567 hybrid_clock.cc:648] HybridClock initialized: now 1786229565131828 us; error 29 us; skew 500 ppm
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.134346 12567 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:52:45.134825 12567 webserver.cc:460] Webserver started at http://127.22.30.1:41607/ using document root <none> and password file <none>
I20260808 22:52:45.135069 12567 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.135165 12567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.136883 12567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.137631 12615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.137821 12567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.137913 12567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "1cf759a6977341b9a9ad0a216f045c24"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "614c22c50f28930de5ec2de2735660d5"
server_key_iv: "9236865903f00955f0cc359bae1a3c95"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.138283 12567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.156210 12567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.158712 12567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.158905 12567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.159170 12567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:45.159699 12629 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:52:45.161057 12567 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:52:45.161129 12567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 22:52:45.161164 12567 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:52:45.162001 12567 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:52:45.162048 12567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 22:52:45.162107 12629 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap starting.
I20260808 22:52:45.169059 12567 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:36837
I20260808 22:52:45.169477 12567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:52:45.169948 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 12567
I20260808 22:52:45.170305 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:45619
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:34085
--webserver_port=39091
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:45.172919 12629 log.cc:824] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:45.173683 12789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:36837 every 8 connection(s)
I20260808 22:52:45.176880 12629 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:52:45.177234 12629 tablet_bootstrap.cc:492] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap complete.
I20260808 22:52:45.178409 12629 ts_tablet_manager.cc:1397] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260808 22:52:45.179737 12629 raft_consensus.cc:348] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:45.180111 12629 raft_consensus.cc:729] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf759a6977341b9a9ad0a216f045c24, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.180292 12629 consensus_queue.cc:260] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } }
I20260808 22:52:45.180578 12629 ts_tablet_manager.cc:1428] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:45.185981 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.173008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:57337 (local address 127.22.30.62:34085)
0808 22:52:45.173208 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.173215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.174040 (+ 825us) server_negotiation.cc:407] Connection header received
0808 22:52:45.174754 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.174761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.174816 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.175826 (+ 1010us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:45.175912 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.177327 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.178820 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.179479 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.180046 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.182176 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:45.182180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:45.182180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:45.182191 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:45.184086 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.184592 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.184595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.184595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.184701 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.184986 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.184987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.184987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.185247 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:45.185335 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.185667 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.185796 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260808 22:52:45.186009 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.172927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:57337)
0808 22:52:45.173872 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.173908 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.174512 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.175003 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.175008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.175419 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:45.175697 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.175703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.177462 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.177463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.178693 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.178696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.179593 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.179594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.179748 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.180352 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:45.180361 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:45.182057 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:45.184176 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:45.184183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:45.184185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:45.184505 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:45.184775 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:45.184776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:45.184776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:45.184908 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:45.185326 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:45.185334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:45.185486 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.185608 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.185785 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":72,"threads_started":1}
I20260808 22:52:45.187204 12790 heartbeater.cc:347] Connected to a master server at 127.22.30.62:34085
I20260808 22:52:45.187335 12790 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:45.187557 12790 heartbeater.cc:511] Master 127.22.30.62:34085 requested a full tablet report, sending...
I20260808 22:52:45.188225 12503 ts_manager.cc:194] Registered new tserver with Master: 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837)
I20260808 22:52:45.189318 12503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:57337
I20260808 22:52:45.190493 12790 heartbeater.cc:503] Master 127.22.30.62:34085 was elected leader, sending a full tablet report...
I20260808 22:52:45.194228 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.189770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36837 (local address 127.0.0.1:45104)
0808 22:52:45.189968 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.190013 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.190058 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.190909 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.190913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.190915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:45.190968 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.190970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.192111 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.192112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.193069 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.193072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.193940 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.193942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.194017 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.194029 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.194105 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.194172 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260808 22:52:45.194235 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.189861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.22.30.1:36837)
0808 22:52:45.190694 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.190698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.190709 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:52:45.190743 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.190745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.190760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.190836 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:45.191048 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.192022 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.193163 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.193842 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.193917 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.193971 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.194128 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.194171 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":71,"threads_started":1}
I20260808 22:52:45.194617 12688 tablet_service.cc:1514] Processing DeleteTablet for tablet 2041c3a8071a45dbb03c959c029a737e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:44 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260808 22:52:45.195080 12802 tablet_replica.cc:330] stopping tablet replica
I20260808 22:52:45.195202 12802 raft_consensus.cc:2229] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:52:45.195276 12802 raft_consensus.cc:2258] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:45.195569 12802 ts_tablet_manager.cc:1905] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:45.197297 12802 ts_tablet_manager.cc:1918] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:52:45.197360 12802 log.cc:1196] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/2041c3a8071a45dbb03c959c029a737e
I20260808 22:52:45.197530 12802 ts_tablet_manager.cc:1939] T 2041c3a8071a45dbb03c959c029a737e P 1cf759a6977341b9a9ad0a216f045c24: Deleting consensus metadata
I20260808 22:52:45.198027 12480 catalog_manager.cc:4841] TS 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837): tablet 2041c3a8071a45dbb03c959c029a737e (table test-table [id=1ee92031ead7412d921f42ea44391842]) successfully deleted
W20260808 22:52:45.253643 12793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.253840 12793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.253880 12793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.254949 12793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.255009 12793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.255052 12793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:52:45.256395 12793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:45619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=39091
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.256667 12793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.256875 12793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.261821 12810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.261835 12809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.262028 12793 server_base.cc:1034] running on GCE node
W20260808 22:52:45.261857 12812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.262264 12793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.262446 12793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:45.263638 12793 hybrid_clock.cc:648] HybridClock initialized: now 1786229565263615 us; error 32 us; skew 500 ppm
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.265969 12793 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:52:45.266408 12793 webserver.cc:460] Webserver started at http://127.22.30.2:39091/ using document root <none> and password file <none>
I20260808 22:52:45.266641 12793 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.266722 12793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.268028 12793 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.268606 12827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.268785 12793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:45.268914 12793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "aa13000474a04c848167062a50b80d38"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "b29b38bdea63c526f773b802586895d4"
server_key_iv: "efa9fd2f52f85e223d156a82f08eea4c"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.269238 12793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.281692 12793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.283277 12793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.283483 12793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.283700 12793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:45.283983 12793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:45.284037 12793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.284083 12793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:45.284112 12793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.290263 12793 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:45619
I20260808 22:52:45.290297 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:45619 every 8 connection(s)
I20260808 22:52:45.290630 12793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:45.296499 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 12793
I20260808 22:52:45.296833 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:37631
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:34085
--webserver_port=39349
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:45.299862 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.291349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:45895)
0808 22:52:45.291519 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.291549 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.291953 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.292175 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.292177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.292664 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:45.292863 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.292867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.293488 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.293489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.294219 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.294221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.294615 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.294616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.294718 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.295219 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:45.295227 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:45.296660 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:45.298347 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:45.298355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:45.298358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:45.298673 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:45.298934 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:45.298935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:45.298935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:45.299045 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:45.299351 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:45.299356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:45.299513 (+ 157us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.299598 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.299713 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260808 22:52:45.299890 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.291347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:45895 (local address 127.22.30.62:34085)
0808 22:52:45.291421 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.291424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.291637 (+ 213us) server_negotiation.cc:407] Connection header received
0808 22:52:45.292044 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.292045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.292059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.292123 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:45.292946 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.293408 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.294289 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.294512 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.294603 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.296759 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:45.296760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:45.296761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:45.296767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:45.298257 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.298763 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.298764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.298765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.298837 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.299116 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.299117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.299118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.299274 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:45.299359 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.299650 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.299756 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:52:45.300360 12986 heartbeater.cc:347] Connected to a master server at 127.22.30.62:34085
I20260808 22:52:45.300472 12986 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:45.300699 12986 heartbeater.cc:511] Master 127.22.30.62:34085 requested a full tablet report, sending...
I20260808 22:52:45.301092 12503 ts_manager.cc:194] Registered new tserver with Master: aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
I20260808 22:52:45.301625 12503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:45895
W20260808 22:52:45.377790 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.377969 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.377990 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.378998 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.379037 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.379065 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:52:45.380434 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:37631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=39349
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.380623 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.380791 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.385726 12997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.385861 12989 server_base.cc:1034] running on GCE node
W20260808 22:52:45.385737 13002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.385901 12998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.386230 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.386425 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:45.387557 12989 hybrid_clock.cc:648] HybridClock initialized: now 1786229565387542 us; error 27 us; skew 500 ppm
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.389864 12989 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:52:45.390308 12989 webserver.cc:460] Webserver started at http://127.22.30.3:39349/ using document root <none> and password file <none>
I20260808 22:52:45.390514 12989 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.390625 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.391953 12989 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.392505 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.392668 12989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:45.392760 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "34c714028e71431b9f7342f66734a676"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "2c2cb8712c58dc24b8fd8de01a190119"
server_key_iv: "0c6390b857ab407bc1f9eb48a1475629"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.393050 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.405962 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.407621 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.407797 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.408002 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:45.408284 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:45.408344 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.408383 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:45.408411 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.414407 12989 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:37631
I20260808 22:52:45.414436 13171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:37631 every 8 connection(s)
I20260808 22:52:45.414744 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:52:45.422526 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.415441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57965 (local address 127.22.30.62:34085)
0808 22:52:45.415490 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.415492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.415667 (+ 175us) server_negotiation.cc:407] Connection header received
0808 22:52:45.416057 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.416058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.416067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.416115 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:45.416872 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.417308 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.418258 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.418445 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.418496 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.420350 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:45.420352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:45.420352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:45.420358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:45.421480 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.421824 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.421825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.421825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.421870 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.422069 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.422069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.422070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.422160 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:45.422207 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.422415 (+ 208us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.422471 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 22:52:45.422721 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.415416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:57965)
0808 22:52:45.415551 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.415585 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.415983 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.416157 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.416160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.416635 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:45.416805 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.416810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.417380 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.417381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.418178 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.418179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.418517 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.418517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.418612 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.418994 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:45.419002 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:45.420270 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:45.421547 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:45.421553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:45.421554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:45.421752 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:45.421926 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:45.421927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:45.421927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:45.422018 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:45.422210 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:45.422214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:45.422320 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.422389 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.422486 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260808 22:52:45.423257 13172 heartbeater.cc:347] Connected to a master server at 127.22.30.62:34085
I20260808 22:52:45.423398 13172 heartbeater.cc:464] Registering TS with master...
I20260808 22:52:45.423564 13172 heartbeater.cc:511] Master 127.22.30.62:34085 requested a full tablet report, sending...
I20260808 22:52:45.423579 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 12989
I20260808 22:52:45.424005 12503 ts_manager.cc:194] Registered new tserver with Master: 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631)
I20260808 22:52:45.424525 12503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:57965
I20260808 22:52:45.429339 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.423898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.22.30.62:34085)
0808 22:52:45.423934 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.423936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.424107 (+ 171us) server_negotiation.cc:407] Connection header received
0808 22:52:45.424156 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.424157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.424166 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.424225 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:45.424915 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.425485 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.426108 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.426257 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.426302 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.427216 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:45.427217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:45.427217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:45.427222 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:45.428452 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.428711 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.428712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.428712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.428752 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:45.428983 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:45.428984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:45.428984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:45.429058 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:45.429158 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.429240 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.429291 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:52:45.430038 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:52:45.432230 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.430356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.22.30.62:34085)
0808 22:52:45.430389 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.430391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.430503 (+ 112us) server_negotiation.cc:407] Connection header received
0808 22:52:45.430555 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.430555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.430578 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.430630 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:52:45.430692 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.431061 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.431696 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.431852 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.431898 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.432017 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:52:45.432086 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:52:45.432136 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.432147 (+ 11us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.432158 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 22:52:45.432461 12503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:52:45.433885 12503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:52:45.435949 12688 tablet_service.cc:1467] Processing CreateTablet for tablet 663b1ade0de84573859a112549173efe (DEFAULT_TABLE table=test-table [id=a23a6cc2a3a04c8e821229fa31743719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:45.436149 12688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663b1ade0de84573859a112549173efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:45.437479 12629 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap starting.
I20260808 22:52:45.437769 12629 tablet_bootstrap.cc:654] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:45.438814 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.435585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:37631 (local address 127.0.0.1:46232)
0808 22:52:45.435632 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.435641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.435679 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.435934 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.435935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.435937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:45.435974 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.435975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.436972 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.436973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.437764 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.437767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.438579 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.438580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.438642 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.438644 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.438708 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.438748 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 22:52:45.438831 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.435647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.22.30.3:37631)
0808 22:52:45.435790 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.435794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.435803 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:52:45.435841 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.435843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.435854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.435935 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:45.436026 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.436885 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.437893 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.438398 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.438486 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.438543 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.438723 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.438756 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260808 22:52:45.439133 12629 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: No bootstrap required, opened a new log
I20260808 22:52:45.439189 12629 ts_tablet_manager.cc:1397] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:52:45.439294 12629 raft_consensus.cc:348] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.439352 12629 raft_consensus.cc:374] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:45.439421 12629 raft_consensus.cc:729] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf759a6977341b9a9ad0a216f045c24, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.439474 12629 consensus_queue.cc:260] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.439574 12629 ts_tablet_manager.cc:1428] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.439620 13087 tablet_service.cc:1467] Processing CreateTablet for tablet 663b1ade0de84573859a112549173efe (DEFAULT_TABLE table=test-table [id=a23a6cc2a3a04c8e821229fa31743719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:45.439940 13087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663b1ade0de84573859a112549173efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:45.441031 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.435633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:45619 (local address 127.0.0.1:48662)
0808 22:52:45.435850 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.435860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.435909 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.437844 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.437846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.437848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:45.437905 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.437906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.438810 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.438811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.439919 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.439922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.440811 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.440812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.440872 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.440876 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.440932 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.440969 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":62,"threads_started":1}
I20260808 22:52:45.441056 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.437525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.22.30.2:45619)
0808 22:52:45.437679 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.437682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.437691 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:52:45.437725 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.437727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.437746 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.437833 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:45.437960 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.438696 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.440004 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.440711 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.440807 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.440860 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.440945 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.440981 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260808 22:52:45.441731 12892 tablet_service.cc:1467] Processing CreateTablet for tablet 663b1ade0de84573859a112549173efe (DEFAULT_TABLE table=test-table [id=a23a6cc2a3a04c8e821229fa31743719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:52:45.442078 12892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663b1ade0de84573859a112549173efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:52:45.443145 13186 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Bootstrap starting.
I20260808 22:52:45.443789 13186 tablet_bootstrap.cc:654] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:45.444211 13186 log.cc:824] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:45.444775 12799 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:45.444857 12799 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.444977 13186 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: No bootstrap required, opened a new log
I20260808 22:52:45.445039 13186 ts_tablet_manager.cc:1397] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:52:45.445117 12799 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
I20260808 22:52:45.445658 13190 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Bootstrap starting.
I20260808 22:52:45.447304 13190 tablet_bootstrap.cc:654] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Neither blocks nor log segments found. Creating new log.
I20260808 22:52:45.447257 13186 raft_consensus.cc:348] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.447496 13186 raft_consensus.cc:374] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:45.447572 13186 raft_consensus.cc:729] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c714028e71431b9f7342f66734a676, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.447722 13186 consensus_queue.cc:260] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.447875 13190 log.cc:824] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:45.448035 13186 ts_tablet_manager.cc:1428] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:52:45.448172 13172 heartbeater.cc:503] Master 127.22.30.62:34085 was elected leader, sending a full tablet report...
I20260808 22:52:45.448608 13190 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: No bootstrap required, opened a new log
I20260808 22:52:45.448674 13190 ts_tablet_manager.cc:1397] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:52:45.448987 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.445252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:45619 (local address 127.22.30.1:36975)
0808 22:52:45.445344 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.445354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.445390 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.445632 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.445633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.445635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:45.445699 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.445700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.446852 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.446853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.448008 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.448012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.448721 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.448722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.448823 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.448826 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.448877 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.448917 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 22:52:45.449132 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.445471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:48245 (local address 127.22.30.3:37631)
0808 22:52:45.445675 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.445676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.445684 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:52:45.445704 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.445704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.445712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.445759 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:45.445963 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.446590 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.448033 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.448754 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.448860 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.448869 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.449049 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.449086 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260808 22:52:45.449208 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.445465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:36975 (local address 127.22.30.2:45619)
0808 22:52:45.445513 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.445515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.445523 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:52:45.445544 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.445544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.445553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.445613 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:52:45.445782 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.446748 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.448122 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.448617 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.448679 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.448688 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.449122 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.449156 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:52:45.449348 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.445319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:37631 (local address 127.22.30.1:48245)
0808 22:52:45.445447 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.445456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.445496 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.445814 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:45.445815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:45.445817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:52:45.445891 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.445893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.446683 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.446684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.447930 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:45.447933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.448883 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.448884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:45.448958 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.448962 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:52:45.449200 (+ 238us) client_negotiation.cc:240] Negotiation successful
0808 22:52:45.449250 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260808 22:52:45.449501 12924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663b1ade0de84573859a112549173efe" candidate_uuid: "1cf759a6977341b9a9ad0a216f045c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa13000474a04c848167062a50b80d38" is_pre_election: true
I20260808 22:52:45.449754 13113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663b1ade0de84573859a112549173efe" candidate_uuid: "1cf759a6977341b9a9ad0a216f045c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c714028e71431b9f7342f66734a676" is_pre_election: true
I20260808 22:52:45.449841 13113 raft_consensus.cc:2454] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf759a6977341b9a9ad0a216f045c24 in term 0.
W20260808 22:52:45.449947 12617 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): Illegal state: must be running to vote when last-logged opid is not known
I20260808 22:52:45.450062 12620 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24, 34c714028e71431b9f7342f66734a676; no voters: aa13000474a04c848167062a50b80d38
I20260808 22:52:45.450157 12799 raft_consensus.cc:2781] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:52:45.450199 12799 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:52:45.450220 12799 raft_consensus.cc:3037] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:45.450399 13190 raft_consensus.cc:348] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.450479 13190 raft_consensus.cc:374] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:52:45.450497 13190 raft_consensus.cc:729] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa13000474a04c848167062a50b80d38, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.450564 13190 consensus_queue.cc:260] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.450793 13190 ts_tablet_manager.cc:1428] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:52:45.450825 12986 heartbeater.cc:503] Master 127.22.30.62:34085 was elected leader, sending a full tablet report...
I20260808 22:52:45.450825 12799 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.450933 12799 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 1 election: Requested vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
I20260808 22:52:45.451124 13113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663b1ade0de84573859a112549173efe" candidate_uuid: "1cf759a6977341b9a9ad0a216f045c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c714028e71431b9f7342f66734a676"
I20260808 22:52:45.451120 12924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663b1ade0de84573859a112549173efe" candidate_uuid: "1cf759a6977341b9a9ad0a216f045c24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa13000474a04c848167062a50b80d38"
I20260808 22:52:45.451186 13113 raft_consensus.cc:3037] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:45.451198 12924 raft_consensus.cc:3037] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:52:45.451874 13113 raft_consensus.cc:2454] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf759a6977341b9a9ad0a216f045c24 in term 1.
I20260808 22:52:45.451946 12924 raft_consensus.cc:2454] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf759a6977341b9a9ad0a216f045c24 in term 1.
I20260808 22:52:45.452049 12620 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24, 34c714028e71431b9f7342f66734a676; no voters:
I20260808 22:52:45.452124 12799 raft_consensus.cc:2781] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:52:45.452239 12799 raft_consensus.cc:686] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf759a6977341b9a9ad0a216f045c24, State: Running, Role: LEADER
I20260808 22:52:45.452303 12799 consensus_queue.cc:237] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.452879 12502 catalog_manager.cc:5495] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "1cf759a6977341b9a9ad0a216f045c24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } health_report { overall_health: HEALTHY } } }
I20260808 22:52:45.459052 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.457111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.22.30.1:36837)
0808 22:52:45.457165 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:45.457166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:45.457179 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:52:45.457203 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:45.457204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:45.457216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:45.457270 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:52:45.457341 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.457796 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.458397 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:45.458586 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:45.458628 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:45.458759 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:52:45.458855 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:52:45.458904 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:45.458953 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 22:52:45.458986 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 22:52:45.460294 13113 raft_consensus.cc:1261] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Refusing update from remote peer 1cf759a6977341b9a9ad0a216f045c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:45.460305 12924 raft_consensus.cc:1261] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Refusing update from remote peer 1cf759a6977341b9a9ad0a216f045c24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:52:45.460569 12799 consensus_queue.cc:1035] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:45.460686 13199 consensus_queue.cc:1035] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:52:45.462638 13202 mvcc.cc:205] Tried to move back new op lower bound from 7316396300123521024 to 7316396300093087744. Current Snapshot: MvccSnapshot[applied={T|T < 7316396300123521024}]
I20260808 22:52:45.462761 13204 mvcc.cc:205] Tried to move back new op lower bound from 7316396300123521024 to 7316396300093087744. Current Snapshot: MvccSnapshot[applied={T|T < 7316396300123521024}]
I20260808 22:52:45.464702 12502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:52:45.464828 12502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:52:45.465428 12502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Sending DeleteTablet for 3 replicas of tablet 663b1ade0de84573859a112549173efe
I20260808 22:52:45.465718 12892 tablet_service.cc:1514] Processing DeleteTablet for tablet 663b1ade0de84573859a112549173efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:45 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:48662
I20260808 22:52:45.465812 12688 tablet_service.cc:1514] Processing DeleteTablet for tablet 663b1ade0de84573859a112549173efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:45 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260808 22:52:45.465885 13087 tablet_service.cc:1514] Processing DeleteTablet for tablet 663b1ade0de84573859a112549173efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:52:45 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46232
I20260808 22:52:45.465916 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 12567
I20260808 22:52:45.471621 13219 tablet_replica.cc:330] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: stopping tablet replica
I20260808 22:52:45.471727 13219 raft_consensus.cc:2229] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:52:45.471825 13219 raft_consensus.cc:2258] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:45.472882 13219 ts_tablet_manager.cc:1905] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:45.475251 13219 ts_tablet_manager.cc:1918] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:52:45.475284 13217 tablet_replica.cc:330] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: stopping tablet replica
I20260808 22:52:45.475325 13219 log.cc:1196] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/663b1ade0de84573859a112549173efe
I20260808 22:52:45.475402 13217 raft_consensus.cc:2229] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:52:45.475507 13217 raft_consensus.cc:2258] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:52:45.475694 13219 ts_tablet_manager.cc:1939] T 663b1ade0de84573859a112549173efe P 34c714028e71431b9f7342f66734a676: Deleting consensus metadata
I20260808 22:52:45.475915 13217 ts_tablet_manager.cc:1905] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:52:45.476074 12481 catalog_manager.cc:4841] TS 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): tablet 663b1ade0de84573859a112549173efe (table test-table [id=a23a6cc2a3a04c8e821229fa31743719]) successfully deleted
I20260808 22:52:45.476946 13217 ts_tablet_manager.cc:1918] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:52:45.477003 13217 log.cc:1196] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/663b1ade0de84573859a112549173efe
I20260808 22:52:45.477275 13217 ts_tablet_manager.cc:1939] T 663b1ade0de84573859a112549173efe P aa13000474a04c848167062a50b80d38: Deleting consensus metadata
I20260808 22:52:45.477630 12479 catalog_manager.cc:4841] TS aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): tablet 663b1ade0de84573859a112549173efe (table test-table [id=a23a6cc2a3a04c8e821229fa31743719]) successfully deleted
W20260808 22:52:45.481258 12480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:36837 (error 108)
I20260808 22:52:45.481453 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 12793
I20260808 22:52:45.481709 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.481613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36837 (local address 127.0.0.1:45118)
0808 22:52:45.481652 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.481680 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:36837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 22:52:45.481807 12480 catalog_manager.cc:4584] TS 1cf759a6977341b9a9ad0a216f045c24 (127.22.30.1:36837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 663b1ade0de84573859a112549173efe: Network error: Client connection negotiation failed: client connection to 127.22.30.1:36837: connect: Connection refused (error 111)
I20260808 22:52:45.496294 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 12989
I20260808 22:52:45.504827 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 12455
I20260808 22:52:45.522030 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34085
--webserver_interface=127.22.30.62
--webserver_port=41649
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:45.598800 13223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.599030 13223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.599068 13223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.600256 13223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:52:45.600317 13223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.600348 13223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.600368 13223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:52:45.600394 13223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:52:45.601800 13223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=41649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.602041 13223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.602231 13223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.606922 13230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.606940 13223 server_base.cc:1034] running on GCE node
W20260808 22:52:45.606923 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.606941 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.607440 13223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.607782 13223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:45.608953 13223 hybrid_clock.cc:648] HybridClock initialized: now 1786229565608937 us; error 48 us; skew 500 ppm
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.611663 13223 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:52:45.612059 13223 webserver.cc:460] Webserver started at http://127.22.30.62:41649/ using document root <none> and password file <none>
I20260808 22:52:45.612267 13223 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.612332 13223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.613593 13223 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.614317 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.614516 13223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.614610 13223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d162b904e57b49c99f7986fd7b60b593"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "090e3725e903397e2efc730ac2bf213b"
server_key_iv: "1607f5ce38744097b55bdbcb4be9c565"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.614945 13223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.631062 13223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.632575 13223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.632714 13223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.636267 13223 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:34085
I20260808 22:52:45.636307 13324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:34085 every 8 connection(s)
I20260808 22:52:45.636656 13223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:52:45.637106 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 13223
I20260808 22:52:45.637476 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:36837
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:34085
--webserver_port=41607
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:45.638725 13325 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:52:45.639595 13325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap starting.
I20260808 22:52:45.649024 13325 log.cc:824] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:45.651084 13325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:52:45.651407 13325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593: Bootstrap complete.
I20260808 22:52:45.653440 13325 raft_consensus.cc:348] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.653630 13325 raft_consensus.cc:729] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d162b904e57b49c99f7986fd7b60b593, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.653780 13325 consensus_queue.cc:260] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.653844 13325 raft_consensus.cc:388] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:52:45.653904 13325 raft_consensus.cc:482] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:52:45.653951 13325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 2 FOLLOWER]: Advancing to term 3
I20260808 22:52:45.654808 13325 raft_consensus.cc:504] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.654886 13325 leader_election.cc:302] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d162b904e57b49c99f7986fd7b60b593; no voters:
I20260808 22:52:45.655063 13325 leader_election.cc:288] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 22:52:45.655110 13329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 22:52:45.655207 13329 raft_consensus.cc:686] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [term 3 LEADER]: Becoming Leader. State: Replica: d162b904e57b49c99f7986fd7b60b593, State: Running, Role: LEADER
I20260808 22:52:45.655313 13325 sys_catalog.cc:564] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:52:45.655359 13329 consensus_queue.cc:237] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } }
I20260808 22:52:45.655709 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d162b904e57b49c99f7986fd7b60b593. Latest consensus state: current_term: 3 leader_uuid: "d162b904e57b49c99f7986fd7b60b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } } }
I20260808 22:52:45.655720 13330 sys_catalog.cc:455] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d162b904e57b49c99f7986fd7b60b593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d162b904e57b49c99f7986fd7b60b593" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34085 } } }
I20260808 22:52:45.655802 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:45.655828 13330 sys_catalog.cc:458] T 00000000000000000000000000000000 P d162b904e57b49c99f7986fd7b60b593 [sys.catalog]: This master's current role is: LEADER
I20260808 22:52:45.656416 13340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:52:45.657183 13340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:52:45.657285 13340 catalog_manager.cc:1259] Loaded cluster ID: 9038df9f33d44d55b34e7146096805fd
I20260808 22:52:45.657344 13340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 22:52:45.657655 13340 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa9901dc9ad google::LogMessage::Fail() at ??:0
@ 0x7fa9901df88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa9901dc4de google::LogMessage::Flush() at ??:0
@ 0x7fa9901dda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9980fd0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa9981068b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa9920aa0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa99209f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa992f036db start_thread at ??:0
@ 0x7fa98ecda61f clone at ??:0
W20260808 22:52:45.735874 13327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.736132 13327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.736167 13327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.737872 13327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.737926 13327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.737965 13327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:52:45.740154 13327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:36837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=41607
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.740386 13327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.740614 13327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.745574 13385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.745628 13383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.745729 13327 server_base.cc:1034] running on GCE node
W20260808 22:52:45.745819 13389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.746043 13327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.746318 13327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:45.747447 13327 hybrid_clock.cc:648] HybridClock initialized: now 1786229565747433 us; error 26 us; skew 500 ppm
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.750377 13327 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:52:45.750782 13327 webserver.cc:460] Webserver started at http://127.22.30.1:41607/ using document root <none> and password file <none>
I20260808 22:52:45.750996 13327 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.751056 13327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.752333 13327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:52:45.752998 13405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.753157 13327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:52:45.753247 13327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "1cf759a6977341b9a9ad0a216f045c24"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "614c22c50f28930de5ec2de2735660d5"
server_key_iv: "9236865903f00955f0cc359bae1a3c95"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.753531 13327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.767704 13327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.769851 13327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.770005 13327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.770224 13327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:45.770712 13416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:52:45.772598 13327 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:52:45.772671 13327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 22:52:45.772713 13327 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:52:45.773259 13327 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:52:45.773315 13327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 22:52:45.773384 13416 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap starting.
I20260808 22:52:45.780098 13327 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:36837
I20260808 22:52:45.780174 13580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:36837 every 8 connection(s)
I20260808 22:52:45.780438 13327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:52:45.784003 13416 log.cc:824] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Log is configured to *not* fsync() on all Append() calls
I20260808 22:52:45.784520 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 13327
I20260808 22:52:45.784857 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:45619
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:34085
--webserver_port=39091
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:52:45.786052 13416 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:52:45.786324 13416 tablet_bootstrap.cc:492] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Bootstrap complete.
I20260808 22:52:45.787752 13416 ts_tablet_manager.cc:1397] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.011s
I20260808 22:52:45.788852 13416 raft_consensus.cc:348] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.789064 13416 raft_consensus.cc:729] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf759a6977341b9a9ad0a216f045c24, State: Initialized, Role: FOLLOWER
I20260808 22:52:45.789223 13416 consensus_queue.cc:260] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:45.789518 13416 ts_tablet_manager.cc:1428] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260808 22:52:45.877902 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:45.878141 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:45.878172 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:45.880275 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:45.880323 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:45.880364 13588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:52:45.882650 13588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:45619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=39091
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:45.883322 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:45.884146 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:45.893221 13600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.893217 13604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:45.893325 13599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:45.893492 13588 server_base.cc:1034] running on GCE node
I20260808 22:52:45.893808 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:45.894089 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260808 22:52:45.894726 13585 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:52:45.781055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:57791)
0808 22:52:45.781222 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.781267 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:45.781958 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:45.894596 (+112638us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260808 22:52:45.894948 13581 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:34085 (0 consecutive failures): Network error: Failed to ping master at 127.22.30.62:34085: Client connection negotiation failed: client connection to 127.22.30.62:34085: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:52:45.895221 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.895134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:57729)
0808 22:52:45.895162 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.895189 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260808 22:52:45.895248 13588 hybrid_clock.cc:648] HybridClock initialized: now 1786229565895231 us; error 31 us; skew 500 ppm
I20260808 22:52:45.895617 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.895498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:33115)
0808 22:52:45.895554 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.895583 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 22:52:45.895742 13581 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 22:52:45 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:45.897662 13588 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:52:45.898073 13588 webserver.cc:460] Webserver started at http://127.22.30.2:39091/ using document root <none> and password file <none>
I20260808 22:52:45.898306 13588 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:45.898351 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:45.899658 13588 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:52:45.900202 13616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.900390 13588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:52:45.900482 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "aa13000474a04c848167062a50b80d38"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "b29b38bdea63c526f773b802586895d4"
server_key_iv: "efa9fd2f52f85e223d156a82f08eea4c"
server_key_version: "encryptionkey@0"
I20260808 22:52:45.900811 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:45.918490 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:45.920333 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:45.920488 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:45.920733 13588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:45.921030 13588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:45.921083 13588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.921113 13588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:45.921144 13588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:45.926934 13588 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:45619
I20260808 22:52:45.926981 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:45619 every 8 connection(s)
I20260808 22:52:45.927265 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:52:45.928191 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.927823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:34605)
0808 22:52:45.927955 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.928054 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
W20260808 22:52:45.928557 13794 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:34085 (0 consecutive failures): Network error: Failed to ping master at 127.22.30.62:34085: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
I20260808 22:52:45.928817 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.928741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:59039)
0808 22:52:45.928779 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.928801 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 22:52:45.929093 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:45.929026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:50771)
0808 22:52:45.929061 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:45.929081 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260808 22:52:45.929231 13794 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:52:45.933403 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 13588
I20260808 22:52:45.933596 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:37631
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:34085
--webserver_port=39349
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:52:46.010905 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:52:46.011119 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:52:46.011155 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:52:46.012238 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:52:46.012305 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:52:46.012344 13798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:52:46.013900 13798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:37631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=39349
--tserver_master_addrs=127.22.30.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:52:46.014149 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:52:46.014375 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:52:46.019153 13808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:46.019245 13798 server_base.cc:1034] running on GCE node
W20260808 22:52:46.019192 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:52:46.019197 13806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:52:46.019604 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:52:46.019892 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:52:46.021026 13798 hybrid_clock.cc:648] HybridClock initialized: now 1786229566021017 us; error 49 us; skew 500 ppm
Aug 08 22:52:46 dist-test-slave-x4qh krb5kdc[11622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229566, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:52:46.023269 13798 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:52:46.023706 13798 webserver.cc:460] Webserver started at http://127.22.30.3:39349/ using document root <none> and password file <none>
I20260808 22:52:46.023931 13798 fs_manager.cc:359] Metadata directory not provided
I20260808 22:52:46.023973 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:52:46.025138 13798 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:52:46.025681 13823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:46.025848 13798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:52:46.025925 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "34c714028e71431b9f7342f66734a676"
format_stamp: "Formatted at 2026-08-08 22:52:44 on dist-test-slave-x4qh"
server_key: "2c2cb8712c58dc24b8fd8de01a190119"
server_key_iv: "0c6390b857ab407bc1f9eb48a1475629"
server_key_version: "encryptionkey@0"
I20260808 22:52:46.026173 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:52:46.050393 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:52:46.052142 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:52:46.052253 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:52:46.052453 13798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:52:46.052691 13798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:52:46.052726 13798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:46.052747 13798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:52:46.052762 13798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:52:46.058933 13798 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:37631
I20260808 22:52:46.058966 14017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:37631 every 8 connection(s)
I20260808 22:52:46.059319 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:52:46.060410 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:46.059986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:52009)
0808 22:52:46.060192 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:46.060292 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
W20260808 22:52:46.060737 14018 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:34085 (0 consecutive failures): Network error: Failed to ping master at 127.22.30.62:34085: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
I20260808 22:52:46.061014 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:46.060934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:34801)
0808 22:52:46.060976 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:46.060999 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 22:52:46.061246 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:46.061201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:50767)
0808 22:52:46.061220 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:46.061235 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260808 22:52:46.061367 14018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:52:46.069357 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 13798
I20260808 22:52:46.896268 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:46.896048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:54097)
0808 22:52:46.896173 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:46.896204 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260808 22:52:46.929679 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:46.929501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:45985)
0808 22:52:46.929606 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:46.929636 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260808 22:52:47.061820 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.061637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:36237)
0808 22:52:47.061736 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:47.061762 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260808 22:52:47.382609 14030 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:47.382728 14030 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:47.383023 14030 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
Aug 08 22:52:47 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.3@KRBTEST.COM
Aug 08 22:52:47 dist-test-slave-x4qh krb5kdc[11622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229565, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.2@KRBTEST.COM
I20260808 22:52:47.394042 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.383313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:40691 (local address 127.22.30.2:45619)
0808 22:52:47.383508 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:47.383520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:47.383561 (+ 41us) server_negotiation.cc:407] Connection header received
0808 22:52:47.383992 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:47.383994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:47.384032 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:47.384140 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:47.385086 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.385884 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.387266 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.387690 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.387807 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:47.390446 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:47.390449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:47.390449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:47.390457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:47.392171 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:47.392634 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:47.392636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:47.392636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:47.392721 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:47.393039 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:47.393040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:47.393040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:47.393366 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:47.393451 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:47.393804 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 22:52:47.393932 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260808 22:52:47.394042 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.383400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:60079 (local address 127.22.30.3:37631)
0808 22:52:47.383545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0808 22:52:47.383556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:52:47.383589 (+ 33us) server_negotiation.cc:407] Connection header received
0808 22:52:47.384006 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:52:47.384008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:52:47.384043 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:52:47.384212 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:52:47.385103 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.386220 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.387221 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.387694 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.387817 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:47.390096 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:52:47.390100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:52:47.390100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:52:47.390128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:52:47.391982 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:47.392612 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:47.392614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:47.392615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:47.392721 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:52:47.393039 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:52:47.393040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:52:47.393040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:52:47.393365 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:52:47.393451 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 22:52:47.393808 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 22:52:47.393933 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260808 22:52:47.394052 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.383191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:45619 (local address 127.22.30.1:40691)
0808 22:52:47.383434 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:47.383444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:47.383501 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:47.384225 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:47.384228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:47.384765 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:47.384973 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:47.384979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.385996 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.385997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:47.387150 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:47.387153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.387781 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.387782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:47.387900 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:47.388510 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:47.388515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:47.390331 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:47.392244 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:47.392244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:47.392245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:47.392532 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:47.392796 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:47.392797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:47.392797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:47.392963 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:47.393446 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:47.393453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:47.393603 (+ 150us) client_negotiation.cc:769] Sending connection context
0808 22:52:47.393737 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 22:52:47.393933 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":130,"thread_start_us":91,"threads_started":1}
I20260808 22:52:47.394110 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.383191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:37631 (local address 127.22.30.1:60079)
0808 22:52:47.383252 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:47.383267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 22:52:47.383346 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:52:47.384225 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:52:47.384228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:52:47.384754 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:52:47.384973 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:47.384979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.386325 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.386327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:47.387145 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:52:47.387147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:52:47.387793 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:52:47.387795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:52:47.387902 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:52:47.388463 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:52:47.388471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:52:47.389941 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:52:47.392081 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:47.392087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:47.392089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:47.392514 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:47.392796 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:52:47.392797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:52:47.392797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:52:47.392963 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:52:47.393446 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:52:47.393453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:52:47.393598 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 22:52:47.393736 (+ 138us) client_negotiation.cc:240] Negotiation successful
0808 22:52:47.393991 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":108}
W20260808 22:52:47.395004 13411 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
W20260808 22:52:47.395588 13407 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
I20260808 22:52:47.395642 13407 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24; no voters: 34c714028e71431b9f7342f66734a676, aa13000474a04c848167062a50b80d38
I20260808 22:52:47.395797 14030 raft_consensus.cc:2726] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:52:47.896837 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.896663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:53387)
0808 22:52:47.896758 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:47.896795 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
I20260808 22:52:47.930202 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:47.930029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:51335)
0808 22:52:47.930140 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:47.930173 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":46,"threads_started":1}
I20260808 22:52:48.062443 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:48.062265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:34793)
0808 22:52:48.062379 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:48.062409 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260808 22:52:48.897579 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:48.897378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:47433)
0808 22:52:48.897506 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:48.897534 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260808 22:52:48.930789 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:48.930608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:38307)
0808 22:52:48.930721 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:48.930753 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260808 22:52:49.021651 14040 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:49.021736 14040 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:49.021972 14040 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
W20260808 22:52:49.022336 13407 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
W20260808 22:52:49.022414 13411 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
I20260808 22:52:49.022490 13411 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24; no voters: 34c714028e71431b9f7342f66734a676, aa13000474a04c848167062a50b80d38
I20260808 22:52:49.022570 14040 raft_consensus.cc:2726] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:52:49.062966 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:49.062791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:55047)
0808 22:52:49.062905 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:49.062935 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260808 22:52:49.898295 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:49.898076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:48559)
0808 22:52:49.898205 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:49.898239 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260808 22:52:49.931329 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:49.931143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:51993)
0808 22:52:49.931260 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:49.931294 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260808 22:52:50.063544 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:50.063316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:36263)
0808 22:52:50.063480 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:50.063511 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260808 22:52:50.898968 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:50.898768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:59877)
0808 22:52:50.898877 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:50.898908 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260808 22:52:50.931874 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:50.931724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:33021)
0808 22:52:50.931818 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:50.931844 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260808 22:52:51.064085 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:51.063913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:54017)
0808 22:52:51.064016 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:51.064049 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260808 22:52:51.899582 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:51.899342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:59319)
0808 22:52:51.899505 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:51.899537 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260808 22:52:51.932426 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:51.932256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:50979)
0808 22:52:51.932370 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:51.932395 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260808 22:52:51.966380 14055 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:51.966446 14055 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:51.966624 14055 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
W20260808 22:52:51.966969 13407 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
W20260808 22:52:51.967020 13411 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
I20260808 22:52:51.967046 13407 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24; no voters: 34c714028e71431b9f7342f66734a676, aa13000474a04c848167062a50b80d38
I20260808 22:52:51.967100 14055 raft_consensus.cc:2726] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:52:52.064759 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:52.064535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:39559)
0808 22:52:52.064674 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:52.064709 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260808 22:52:52.900172 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:52.899988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:41747)
0808 22:52:52.900098 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:52.900128 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260808 22:52:52.932900 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:52.932750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:40743)
0808 22:52:52.932846 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:52.932872 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
I20260808 22:52:53.065349 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:53.065172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:48213)
0808 22:52:53.065291 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:53.065317 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260808 22:52:53.900790 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:53.900593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:37913)
0808 22:52:53.900711 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:53.900744 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260808 22:52:53.933461 14063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:53.933270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:58051)
0808 22:52:53.933397 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:53.933427 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260808 22:52:54.065939 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:54.065759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:49361)
0808 22:52:54.065874 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:54.065906 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260808 22:52:54.901593 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:54.901394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:44657)
0808 22:52:54.901515 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:54.901548 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260808 22:52:54.934123 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:54.933935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:52317)
0808 22:52:54.934050 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:54.934083 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260808 22:52:55.066519 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:55.066335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:34913)
0808 22:52:55.066450 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:55.066482 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260808 22:52:55.148875 14073 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:52:55.148998 14073 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:52:55.149214 14073 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
W20260808 22:52:55.149605 13411 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
W20260808 22:52:55.149708 13407 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
I20260808 22:52:55.149801 13407 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24; no voters: 34c714028e71431b9f7342f66734a676, aa13000474a04c848167062a50b80d38
I20260808 22:52:55.149880 14073 raft_consensus.cc:2726] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:52:55.902212 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:55.902009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:40423)
0808 22:52:55.902135 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:55.902168 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260808 22:52:55.934790 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:55.934609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:47701)
0808 22:52:55.934723 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:55.934751 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260808 22:52:56.067178 14078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:56.066969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:54079)
0808 22:52:56.067074 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:56.067115 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260808 22:52:56.902846 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:56.902616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:59055)
0808 22:52:56.902767 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:56.902802 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260808 22:52:56.935468 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:56.935207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:51369)
0808 22:52:56.935339 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:56.935412 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260808 22:52:57.067876 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:57.067641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:34241)
0808 22:52:57.067819 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:57.067844 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260808 22:52:57.903625 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:57.903353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:52367)
0808 22:52:57.903543 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:57.903576 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":133,"threads_started":1}
I20260808 22:52:57.936012 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:57.935822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:54141)
0808 22:52:57.935953 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:57.935978 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260808 22:52:58.068393 14087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:58.068217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:32787)
0808 22:52:58.068329 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:58.068360 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260808 22:52:58.904227 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:58.904019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:54789)
0808 22:52:58.904152 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:58.904184 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260808 22:52:58.936606 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:58.936421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:56983)
0808 22:52:58.936525 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:58.936555 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260808 22:52:59.069017 14091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:59.068820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:54965)
0808 22:52:59.068944 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:59.068977 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260808 22:52:59.904834 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:59.904641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:60931)
0808 22:52:59.904758 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:59.904789 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260808 22:52:59.937230 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:52:59.936991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:53901)
0808 22:52:59.937166 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 22:52:59.937199 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":50,"threads_started":1}
I20260808 22:53:00.069602 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:00.069413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:59191)
0808 22:53:00.069527 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:00.069571 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":50,"threads_started":1}
I20260808 22:53:00.302074 14097 raft_consensus.cc:482] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:00.302183 14097 raft_consensus.cc:504] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c714028e71431b9f7342f66734a676" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 37631 } } peers { permanent_uuid: "aa13000474a04c848167062a50b80d38" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 45619 } } peers { permanent_uuid: "1cf759a6977341b9a9ad0a216f045c24" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36837 } }
I20260808 22:53:00.302357 14097 leader_election.cc:288] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631), aa13000474a04c848167062a50b80d38 (127.22.30.2:45619)
W20260808 22:53:00.302711 13411 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 34c714028e71431b9f7342f66734a676 (127.22.30.3:37631): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
W20260808 22:53:00.302824 13407 leader_election.cc:341] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa13000474a04c848167062a50b80d38 (127.22.30.2:45619): Not found: Tablet not found: 663b1ade0de84573859a112549173efe
I20260808 22:53:00.302886 13407 leader_election.cc:302] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cf759a6977341b9a9ad0a216f045c24; no voters: 34c714028e71431b9f7342f66734a676, aa13000474a04c848167062a50b80d38
I20260808 22:53:00.302968 14097 raft_consensus.cc:2726] T 663b1ade0de84573859a112549173efe P 1cf759a6977341b9a9ad0a216f045c24 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:00.905465 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:00.905278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.1:46281)
0808 22:53:00.905394 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:00.905423 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260808 22:53:00.937958 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:00.937716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.2:49941)
0808 22:53:00.937890 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:00.937919 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":53,"threads_started":1}
I20260808 22:53:01.070184 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:01.069977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:34085 (local address 127.22.30.3:40623)
0808 22:53:01.070113 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:01.070149 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:34085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260808 22:53:01.079679 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 13327
I20260808 22:53:01.091794 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 13588
I20260808 22:53:01.098152 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 13798
2026-08-08T22:53:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18287 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:53:01 dist-test-slave-x4qh krb5kdc[14109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:53:01 dist-test-slave-x4qh krb5kdc[14109](info): set up 2 sockets
Aug 08 22:53:01 dist-test-slave-x4qh krb5kdc[14109](info): commencing operation
krb5kdc: starting...
W20260808 22:53:02.263132 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:53:02.375329 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38615
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:02.451246 14133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:02.451532 14133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:02.451570 14133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:02.452581 14133 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 22:53:02.452639 14133 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 22:53:02.452659 14133 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 22:53:02.452675 14133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:53:02.452701 14133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:02.452731 14133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:02.452754 14133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:53:02.452778 14133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:53:02.454088 14133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:02.454348 14133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:02.454548 14133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:02.459411 14141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:02.459506 14145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.459448 14133 server_base.cc:1034] running on GCE node
W20260808 22:53:02.459506 14142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.459822 14133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:02.460109 14133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:02.461266 14133 hybrid_clock.cc:648] HybridClock initialized: now 1786229582461262 us; error 36 us; skew 500 ppm
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:02.463589 14133 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:53:02.464038 14133 webserver.cc:460] Webserver started at http://127.22.30.62:44115/ using document root <none> and password file <none>
I20260808 22:53:02.464255 14133 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:02.464337 14133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:02.464464 14133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:53:02.465363 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "c8d130379a7f4657ab2e0d3c9263b4b7"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ebc18c443f25ca0583182d26023064f2"
server_key_iv: "9f315fb0d9191214f3fbde14ece50993"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.465705 14133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "c8d130379a7f4657ab2e0d3c9263b4b7"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ebc18c443f25ca0583182d26023064f2"
server_key_iv: "9f315fb0d9191214f3fbde14ece50993"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.466933 14133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:53:02.467636 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.467793 14133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:53:02.467885 14133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "c8d130379a7f4657ab2e0d3c9263b4b7"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ebc18c443f25ca0583182d26023064f2"
server_key_iv: "9f315fb0d9191214f3fbde14ece50993"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.467960 14133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:02.486475 14133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:02.488087 14133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:02.488245 14133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:02.492451 14133 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:38615
I20260808 22:53:02.492502 14251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:38615 every 8 connection(s)
I20260808 22:53:02.492835 14133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:53:02.493642 14252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:02.495710 14252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap starting.
I20260808 22:53:02.496237 14252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:02.496544 14252 log.cc:824] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:02.497145 14252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: No bootstrap required, opened a new log
I20260808 22:53:02.498813 14252 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:02.498936 14252 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:02.498980 14252 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d130379a7f4657ab2e0d3c9263b4b7, State: Initialized, Role: FOLLOWER
I20260808 22:53:02.499127 14252 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:02.499183 14252 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:53:02.499221 14252 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:53:02.499272 14252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:02.499831 14252 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:02.499948 14252 leader_election.cc:302] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d130379a7f4657ab2e0d3c9263b4b7; no voters:
I20260808 22:53:02.500083 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 14133
I20260808 22:53:02.500126 14252 leader_election.cc:288] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:53:02.500175 14255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:53:02.500197 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:53:02.500346 14255 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 LEADER]: Becoming Leader. State: Replica: c8d130379a7f4657ab2e0d3c9263b4b7, State: Running, Role: LEADER
I20260808 22:53:02.500378 14252 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:53:02.500444 22648 external_mini_cluster.cc:1428] Setting key c1eba66e150fe02fa932070c281a4ed8
I20260808 22:53:02.500490 14255 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:02.502103 14256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d130379a7f4657ab2e0d3c9263b4b7. Latest consensus state: current_term: 1 leader_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } } }
I20260808 22:53:02.502252 14258 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } } }
I20260808 22:53:02.502434 14256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:53:02.502447 14258 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: This master's current role is: LEADER
W20260808 22:53:02.502705 14278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:53:02.502756 14278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:53:02.502807 14283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:53:02.502954 14283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:53:02.504226 14283 catalog_manager.cc:1347] Generated new cluster ID: 3eca85c3dcc24cae9a7c0f6c98c10e78
I20260808 22:53:02.504272 14283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:02.510514 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.500875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.22.30.62:38615)
0808 22:53:02.501037 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:02.501047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:02.501102 (+ 55us) server_negotiation.cc:407] Connection header received
0808 22:53:02.501206 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:02.501213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:02.501260 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:02.501444 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:02.502197 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.503480 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.504306 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.504691 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.504800 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.507027 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:02.507031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:02.507031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:02.507041 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:02.509079 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.509422 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.509425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.509425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.509514 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.509721 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.509722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.509722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.509935 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:02.510064 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:02.510179 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 22:53:02.510340 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260808 22:53:02.520867 14283 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:53:02.521502 14283 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:53:02.528254 14283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Generated new TSK 0
I20260808 22:53:02.528403 14283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:53:02.554097 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38615
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:02.630152 14291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:02.630398 14291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:02.630436 14291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:02.631596 14291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:02.631652 14291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:02.631704 14291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:53:02.632956 14291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:02.633183 14291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:02.633396 14291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:02.637990 14304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:02.638021 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.638048 14291 server_base.cc:1034] running on GCE node
W20260808 22:53:02.638016 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.638407 14291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:02.638645 14291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:02.639806 14291 hybrid_clock.cc:648] HybridClock initialized: now 1786229582639789 us; error 30 us; skew 500 ppm
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:02.642153 14291 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:53:02.642521 14291 webserver.cc:460] Webserver started at http://127.22.30.1:38071/ using document root <none> and password file <none>
I20260808 22:53:02.642717 14291 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:02.642797 14291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:02.642927 14291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:53:02.643915 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "d843968709ce4395bad70fec8f2e5a8f"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ca6a34f859844dbcf58a66e079f11ac2"
server_key_iv: "878ff6788e643e5a14f99022d4648446"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.644254 14291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "d843968709ce4395bad70fec8f2e5a8f"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ca6a34f859844dbcf58a66e079f11ac2"
server_key_iv: "878ff6788e643e5a14f99022d4648446"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.645561 14291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:53:02.646239 14312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.646397 14291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:53:02.646488 14291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "d843968709ce4395bad70fec8f2e5a8f"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ca6a34f859844dbcf58a66e079f11ac2"
server_key_iv: "878ff6788e643e5a14f99022d4648446"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.646560 14291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:02.660964 14291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:02.662528 14291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:02.662673 14291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:02.662880 14291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:02.663142 14291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:02.663192 14291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.663228 14291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:02.663257 14291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.668717 14291 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:35677
I20260808 22:53:02.668767 14484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:35677 every 8 connection(s)
I20260808 22:53:02.669042 14291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:02.677547 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.669679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:42087 (local address 127.22.30.62:38615)
0808 22:53:02.669737 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:02.669740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:02.669934 (+ 194us) server_negotiation.cc:407] Connection header received
0808 22:53:02.670503 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:02.670505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:02.670520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:02.670581 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:02.671641 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.672138 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.673029 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.673224 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.673296 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.675248 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:02.675249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:02.675249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:02.675258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:02.676455 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.676800 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.676800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.676801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.676849 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.677046 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.677046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.677047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.677138 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:02.677203 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:02.677423 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 22:53:02.677486 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:53:02.677700 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.669660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:42087)
0808 22:53:02.669803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:02.669864 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:02.670400 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:02.670659 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:02.670663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:02.671229 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:02.671551 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:02.671556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.672229 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.672230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:02.672944 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:02.672946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.673308 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.673309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:02.673432 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.673833 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:02.673840 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:02.675163 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:02.676526 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:02.676532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:02.676533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:02.676729 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:02.676905 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:02.676906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:02.676906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:02.676994 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:02.677208 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:02.677212 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:02.677316 (+ 104us) client_negotiation.cc:769] Sending connection context
0808 22:53:02.677393 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 22:53:02.677500 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260808 22:53:02.678241 14485 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38615
I20260808 22:53:02.678323 14485 heartbeater.cc:464] Registering TS with master...
I20260808 22:53:02.678504 14485 heartbeater.cc:511] Master 127.22.30.62:38615 requested a full tablet report, sending...
I20260808 22:53:02.678866 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 14291
I20260808 22:53:02.678974 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:53:02.679071 14184 ts_manager.cc:194] Registered new tserver with Master: d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677)
I20260808 22:53:02.679195 22648 external_mini_cluster.cc:1428] Setting key e0401ed273ae6796dfa04cca53db30e8
I20260808 22:53:02.679752 14184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:42087
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:53:02.701658 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38615
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:02.778977 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:02.779143 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:02.779165 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:02.780231 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:02.780270 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:02.780349 14494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:53:02.781798 14494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:02.782073 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:02.782306 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:02.787168 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.787230 14494 server_base.cc:1034] running on GCE node
W20260808 22:53:02.787153 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:02.787153 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.787649 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:02.787876 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:02.789011 14494 hybrid_clock.cc:648] HybridClock initialized: now 1786229582788998 us; error 27 us; skew 500 ppm
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:02.791363 14494 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:53:02.791801 14494 webserver.cc:460] Webserver started at http://127.22.30.2:32837/ using document root <none> and password file <none>
I20260808 22:53:02.792035 14494 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:02.792102 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:02.792255 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:53:02.793224 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "3db1b2d6649143d4baca7f6702fab7a1"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "db357d8cfbf016f3763199baf2623a2d"
server_key_iv: "8cae16332d578636fb439639fb6181e1"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.793576 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "3db1b2d6649143d4baca7f6702fab7a1"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "db357d8cfbf016f3763199baf2623a2d"
server_key_iv: "8cae16332d578636fb439639fb6181e1"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.794859 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:53:02.795596 14514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.795765 14494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:53:02.795866 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "3db1b2d6649143d4baca7f6702fab7a1"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "db357d8cfbf016f3763199baf2623a2d"
server_key_iv: "8cae16332d578636fb439639fb6181e1"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.795948 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:02.816174 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:02.817898 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:02.818076 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:02.818302 14494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:02.818606 14494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:02.818662 14494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.818702 14494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:02.818732 14494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.824839 14494 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:43235
I20260808 22:53:02.824877 14685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:43235 every 8 connection(s)
I20260808 22:53:02.825191 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:53:02.826333 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 14494
I20260808 22:53:02.826411 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:53:02.826689 22648 external_mini_cluster.cc:1428] Setting key f11f57a6d1da3cd95c1bb390d8481007
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:02.835664 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.825936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:33167 (local address 127.22.30.62:38615)
0808 22:53:02.825999 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:02.826006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:02.826142 (+ 136us) server_negotiation.cc:407] Connection header received
0808 22:53:02.826690 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:02.826692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:02.826705 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:02.826790 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:02.827887 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.828630 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.829969 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.830157 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.830211 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.832612 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:02.832614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:02.832615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:02.832623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:02.834284 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.834672 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.834673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.834674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.834738 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.834999 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.835000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.835001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.835123 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:02.835196 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:02.835494 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 22:53:02.835577 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 22:53:02.835850 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.825890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:33167)
0808 22:53:02.826031 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:02.826067 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:02.826562 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:02.826873 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:02.826876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:02.827608 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:02.827791 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:02.827796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.828723 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.828723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:02.829881 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:02.829884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.830243 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.830244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:02.830410 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.830843 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:02.830852 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:02.832517 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:02.834366 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:02.834372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:02.834373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:02.834583 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:02.834817 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:02.834818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:02.834818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:02.834920 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:02.835187 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:02.835194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:02.835311 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 22:53:02.835469 (+ 158us) client_negotiation.cc:240] Negotiation successful
0808 22:53:02.835619 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260808 22:53:02.836288 14686 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38615
I20260808 22:53:02.836390 14686 heartbeater.cc:464] Registering TS with master...
I20260808 22:53:02.836565 14686 heartbeater.cc:511] Master 127.22.30.62:38615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
I20260808 22:53:02.836934 14184 ts_manager.cc:194] Registered new tserver with Master: 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235)
I20260808 22:53:02.837275 14184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:33167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:53:02.852245 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38615
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:02.930887 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:02.931062 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:02.931085 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:02.932219 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:02.932260 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:02.932287 14693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:53:02.933624 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:02.933809 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:02.933965 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:02.938772 14701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:02.938783 14702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:02.938781 14706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:02.938912 14693 server_base.cc:1034] running on GCE node
I20260808 22:53:02.939201 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:02.939467 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:02.940603 14693 hybrid_clock.cc:648] HybridClock initialized: now 1786229582940588 us; error 37 us; skew 500 ppm
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:02.942885 14693 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:53:02.943228 14693 webserver.cc:460] Webserver started at http://127.22.30.3:46555/ using document root <none> and password file <none>
I20260808 22:53:02.943476 14693 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:02.943523 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:02.943607 14693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:53:02.944502 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "8221083b44be4364b1f1d44273d0effb"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "8a3dbc8925bcef58d17952ca43b9e971"
server_key_iv: "4989ccc0e57e692e869439871b1d4989"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.944785 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "8221083b44be4364b1f1d44273d0effb"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "8a3dbc8925bcef58d17952ca43b9e971"
server_key_iv: "4989ccc0e57e692e869439871b1d4989"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.946020 14693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:53:02.946784 14715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.946934 14693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:53:02.947017 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "8221083b44be4364b1f1d44273d0effb"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "8a3dbc8925bcef58d17952ca43b9e971"
server_key_iv: "4989ccc0e57e692e869439871b1d4989"
server_key_version: "encryptionkey@0"
I20260808 22:53:02.947069 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:02.969827 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:02.971642 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:02.971742 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:02.971889 14693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:02.972105 14693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:02.972132 14693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.972151 14693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:02.972164 14693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:02.977278 14693 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:33591
I20260808 22:53:02.977342 14903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:33591 every 8 connection(s)
I20260808 22:53:02.977602 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:53:02 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229582, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:02.986060 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.978226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:50079 (local address 127.22.30.62:38615)
0808 22:53:02.978292 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:02.978295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:02.978475 (+ 180us) server_negotiation.cc:407] Connection header received
0808 22:53:02.978844 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:02.978844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:02.978854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:02.978902 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:02.979794 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.980259 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.981189 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.981363 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.981410 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.983329 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:02.983331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:02.983331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:02.983337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:02.984755 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.985144 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.985145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.985146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.985203 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.985441 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.985442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.985442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.985548 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:02.985616 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:02.985908 (+ 292us) server_negotiation.cc:299] Negotiation successful
0808 22:53:02.985977 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:53:02.986233 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.978206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:50079)
0808 22:53:02.978368 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:02.978397 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:02.978763 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:02.978969 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:02.978972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:02.979537 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:02.979703 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:02.979708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.980340 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.980340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:02.981115 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:02.981117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.981432 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.981433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:02.981526 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.981948 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:02.981956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:02.983228 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:02.984855 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:02.984862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:02.984864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:02.985067 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:02.985271 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:02.985272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:02.985272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:02.985372 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:02.985621 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:02.985628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:02.985751 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 22:53:02.985874 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 22:53:02.986019 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260808 22:53:02.986759 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 14693
I20260808 22:53:02.986824 14906 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38615
I20260808 22:53:02.986907 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:53:02.986914 14906 heartbeater.cc:464] Registering TS with master...
I20260808 22:53:02.987085 14906 heartbeater.cc:511] Master 127.22.30.62:38615 requested a full tablet report, sending...
I20260808 22:53:02.987123 22648 external_mini_cluster.cc:1428] Setting key a01796a30f96c572fb5378e06993c35b
I20260808 22:53:02.987543 14183 ts_manager.cc:194] Registered new tserver with Master: 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
I20260808 22:53:02.988039 14183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:50079
I20260808 22:53:02.988078 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:53:02.995113 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.989518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.22.30.62:38615)
0808 22:53:02.989587 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:02.989589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:02.989690 (+ 101us) server_negotiation.cc:407] Connection header received
0808 22:53:02.989763 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:02.989763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:02.989772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:02.989826 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:02.990421 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.990861 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.991671 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:02.991840 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:02.991911 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:02.992908 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:02.992909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:02.992909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:02.992915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:02.993995 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.994342 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.994343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.994343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.994409 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:02.994664 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:02.994665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:02.994665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:02.994783 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:02.994873 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:02.994970 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 22:53:02.995033 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:53:02.996446 14183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:53:02.997874 14183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:53:03.003782 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.000049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.22.30.3:33591)
0808 22:53:03.000248 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:03.000252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:03.000261 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:53:03.000301 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:03.000303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:03.000316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:03.000429 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:03.000613 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.001490 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.002686 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.003330 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.003452 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.003509 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:03.003659 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 22:53:03.003693 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260808 22:53:03.003773 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.999834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:33591 (local address 127.0.0.1:53252)
0808 22:53:02.999928 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:02.999959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:03.000014 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:03.000465 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:03.000468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:03.000471 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:03.000533 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.000534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.001611 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.001612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.002598 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.002600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.003457 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.003458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.003536 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.003548 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:53:03.003641 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 22:53:03.003693 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":55,"threads_started":1}
I20260808 22:53:03.004415 14802 tablet_service.cc:1467] Processing CreateTablet for tablet 47e420296e6b4d23a5c882cee97d3078 (DEFAULT_TABLE table=test-table [id=56e17ee2fa474f57a844bf5b323002ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:53:03.004732 14802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e420296e6b4d23a5c882cee97d3078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:03.004861 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.999855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35677 (local address 127.0.0.1:40842)
0808 22:53:03.000228 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:03.000237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:03.000285 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:03.000761 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:03.000762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:03.000764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:03.000811 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.000813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.002406 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.002407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.003285 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.003288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.004603 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.004604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.004666 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.004670 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:53:03.004724 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 22:53:03.004763 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260808 22:53:03.005002 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:02.999855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43235 (local address 127.0.0.1:46046)
0808 22:53:03.000050 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:03.000060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:03.000114 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:03.000863 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:03.000864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:03.000866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:03.000907 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.000909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.002209 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.002210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.003187 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.003189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.004751 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.004752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.004826 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.004830 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:53:03.004883 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:53:03.004923 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":50,"threads_started":1}
I20260808 22:53:03.005261 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.000062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.22.30.2:43235)
0808 22:53:03.000514 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:03.000518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:03.000527 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:53:03.000589 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:03.000591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:03.000690 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:03.000779 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:03.000992 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.002125 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.003286 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.004667 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.004736 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.005120 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:03.005145 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 22:53:03.005183 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":82,"threads_started":1}
I20260808 22:53:03.005517 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.000263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.22.30.1:35677)
0808 22:53:03.000495 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:03.000499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:03.000509 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:53:03.000609 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:03.000614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:03.000632 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:03.000891 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:03.000969 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.002296 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.003941 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.004479 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.004570 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.004625 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:03.005303 (+ 678us) server_negotiation.cc:299] Negotiation successful
0808 22:53:03.005461 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
I20260808 22:53:03.005893 14587 tablet_service.cc:1467] Processing CreateTablet for tablet 47e420296e6b4d23a5c882cee97d3078 (DEFAULT_TABLE table=test-table [id=56e17ee2fa474f57a844bf5b323002ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:53:03.006093 14380 tablet_service.cc:1467] Processing CreateTablet for tablet 47e420296e6b4d23a5c882cee97d3078 (DEFAULT_TABLE table=test-table [id=56e17ee2fa474f57a844bf5b323002ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:53:03.006206 14587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e420296e6b4d23a5c882cee97d3078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:03.006392 14380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47e420296e6b4d23a5c882cee97d3078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:03.007829 14931 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Bootstrap starting.
I20260808 22:53:03.008497 14931 tablet_bootstrap.cc:654] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:03.009006 14933 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Bootstrap starting.
I20260808 22:53:03.009027 14931 log.cc:824] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:03.009511 14934 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap starting.
I20260808 22:53:03.009723 14933 tablet_bootstrap.cc:654] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:03.009791 14931 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: No bootstrap required, opened a new log
I20260808 22:53:03.009858 14931 ts_tablet_manager.cc:1397] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:53:03.010175 14934 tablet_bootstrap.cc:654] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:03.010344 14933 log.cc:824] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:03.010810 14934 log.cc:824] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:03.011158 14933 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: No bootstrap required, opened a new log
I20260808 22:53:03.011224 14933 ts_tablet_manager.cc:1397] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:53:03.011585 14934 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: No bootstrap required, opened a new log
I20260808 22:53:03.011662 14934 ts_tablet_manager.cc:1397] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:53:03.011904 14931 raft_consensus.cc:348] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.012063 14931 raft_consensus.cc:374] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:03.012099 14931 raft_consensus.cc:729] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221083b44be4364b1f1d44273d0effb, State: Initialized, Role: FOLLOWER
I20260808 22:53:03.012218 14931 consensus_queue.cc:260] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.012463 14931 ts_tablet_manager.cc:1428] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:53:03.012573 14906 heartbeater.cc:503] Master 127.22.30.62:38615 was elected leader, sending a full tablet report...
I20260808 22:53:03.013055 14933 raft_consensus.cc:348] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.013172 14933 raft_consensus.cc:374] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:03.013191 14933 raft_consensus.cc:729] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db1b2d6649143d4baca7f6702fab7a1, State: Initialized, Role: FOLLOWER
I20260808 22:53:03.013302 14933 consensus_queue.cc:260] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.013397 14934 raft_consensus.cc:348] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.013563 14934 raft_consensus.cc:374] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:03.013590 14933 ts_tablet_manager.cc:1428] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 22:53:03.013603 14934 raft_consensus.cc:729] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843968709ce4395bad70fec8f2e5a8f, State: Initialized, Role: FOLLOWER
I20260808 22:53:03.013751 14686 heartbeater.cc:503] Master 127.22.30.62:38615 was elected leader, sending a full tablet report...
I20260808 22:53:03.013716 14934 consensus_queue.cc:260] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.014002 14934 ts_tablet_manager.cc:1428] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:53:03.014004 14485 heartbeater.cc:503] Master 127.22.30.62:38615 was elected leader, sending a full tablet report...
W20260808 22:53:03.075737 14688 tablet.cc:2369] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:53:03.151816 14941 raft_consensus.cc:482] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:03.151942 14941 raft_consensus.cc:504] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.152331 14941 leader_election.cc:288] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677)
I20260808 22:53:03.154886 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.152506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:55449 (local address 127.22.30.2:43235)
0808 22:53:03.152572 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:03.152575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:03.152609 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:53:03.152678 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:03.152679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:03.152691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:03.152753 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:03.152928 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.153416 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.154224 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.154569 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.154617 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.154624 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:03.154787 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 22:53:03.154828 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:53:03.154881 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.152464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43235 (local address 127.22.30.3:55449)
0808 22:53:03.152557 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:03.152568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:03.152620 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:03.152779 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:03.152780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:03.152782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:03.152854 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.152856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.153486 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.153486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.154138 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.154139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.154640 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.154641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.154712 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.154715 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:53:03.154763 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 22:53:03.154793 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 22:53:03.154947 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.152596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:41351 (local address 127.22.30.1:35677)
0808 22:53:03.152634 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:03.152636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:03.152792 (+ 156us) server_negotiation.cc:407] Connection header received
0808 22:53:03.152820 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:03.152822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:03.152836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:03.152892 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:03.153049 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.153478 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.154275 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.154584 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.154664 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.154673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:03.154840 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 22:53:03.154883 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:53:03.154963 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.152501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35677 (local address 127.22.30.3:41351)
0808 22:53:03.152707 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:03.152716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:03.152769 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:03.152921 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:03.152922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:03.152923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:03.152984 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.152985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.153561 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.153562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.154192 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:03.154194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.154683 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.154684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:03.154759 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.154763 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:53:03.154816 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:53:03.154847 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
I20260808 22:53:03.155299 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e420296e6b4d23a5c882cee97d3078" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d843968709ce4395bad70fec8f2e5a8f" is_pre_election: true
I20260808 22:53:03.155304 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e420296e6b4d23a5c882cee97d3078" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db1b2d6649143d4baca7f6702fab7a1" is_pre_election: true
I20260808 22:53:03.155443 14616 raft_consensus.cc:2454] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 0.
I20260808 22:53:03.155460 14412 raft_consensus.cc:2454] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 0.
I20260808 22:53:03.155640 14718 leader_election.cc:302] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221083b44be4364b1f1d44273d0effb, d843968709ce4395bad70fec8f2e5a8f; no voters:
I20260808 22:53:03.155805 14941 raft_consensus.cc:2781] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:53:03.155874 14941 raft_consensus.cc:482] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:53:03.155905 14941 raft_consensus.cc:3037] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:03.156538 14941 raft_consensus.cc:504] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.156692 14941 leader_election.cc:288] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 election: Requested vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677)
I20260808 22:53:03.156824 14616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e420296e6b4d23a5c882cee97d3078" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db1b2d6649143d4baca7f6702fab7a1"
I20260808 22:53:03.156852 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47e420296e6b4d23a5c882cee97d3078" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d843968709ce4395bad70fec8f2e5a8f"
I20260808 22:53:03.156903 14616 raft_consensus.cc:3037] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:03.156908 14412 raft_consensus.cc:3037] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:03.157740 14412 raft_consensus.cc:2454] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 1.
I20260808 22:53:03.157740 14616 raft_consensus.cc:2454] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 1.
I20260808 22:53:03.157902 14718 leader_election.cc:302] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221083b44be4364b1f1d44273d0effb, d843968709ce4395bad70fec8f2e5a8f; no voters:
I20260808 22:53:03.157994 14941 raft_consensus.cc:2781] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:53:03.158111 14941 raft_consensus.cc:686] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 1 LEADER]: Becoming Leader. State: Replica: 8221083b44be4364b1f1d44273d0effb, State: Running, Role: LEADER
I20260808 22:53:03.158229 14941 consensus_queue.cc:237] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:03.158973 14183 catalog_manager.cc:5495] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221083b44be4364b1f1d44273d0effb (127.22.30.3). New cstate: current_term: 1 leader_uuid: "8221083b44be4364b1f1d44273d0effb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } health_report { overall_health: UNKNOWN } } }
W20260808 22:53:03.169721 14489 tablet.cc:2369] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:53:03.183310 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.181270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.22.30.3:33591)
0808 22:53:03.181332 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:03.181334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:03.181340 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:53:03.181364 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:03.181365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:03.181379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:03.181431 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:53:03.181583 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.181994 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.182680 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:03.182846 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:03.182892 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:03.183026 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:53:03.183106 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:53:03.183162 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:03.183200 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 22:53:03.183215 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 22:53:03.184986 14616 raft_consensus.cc:1261] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Refusing update from remote peer 8221083b44be4364b1f1d44273d0effb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:53:03.184989 14412 raft_consensus.cc:1261] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Refusing update from remote peer 8221083b44be4364b1f1d44273d0effb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:53:03.185333 14941 consensus_queue.cc:1035] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:53:03.185483 14947 consensus_queue.cc:1035] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:53:03.187870 14952 mvcc.cc:205] Tried to move back new op lower bound from 7316396372723154944 to 7316396372616704000. Current Snapshot: MvccSnapshot[applied={T|T < 7316396372723154944}]
I20260808 22:53:03.189894 14184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:53:03.189993 14184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:53:03.190768 14184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Sending DeleteTablet for 3 replicas of tablet 47e420296e6b4d23a5c882cee97d3078
I20260808 22:53:03.191076 14802 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e420296e6b4d23a5c882cee97d3078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:53:03 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:53252
I20260808 22:53:03.191185 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 14291
I20260808 22:53:03.191172 14587 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e420296e6b4d23a5c882cee97d3078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:53:03 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:46046
I20260808 22:53:03.191286 14380 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e420296e6b4d23a5c882cee97d3078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:53:03 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:40842
I20260808 22:53:03.193640 14968 tablet_replica.cc:330] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: stopping tablet replica
I20260808 22:53:03.193760 14968 raft_consensus.cc:2229] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:53:03.193876 14968 raft_consensus.cc:2258] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:53:03.195844 14968 ts_tablet_manager.cc:1905] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:53:03.198643 14968 ts_tablet_manager.cc:1918] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:53:03.198786 14968 log.cc:1196] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/47e420296e6b4d23a5c882cee97d3078
I20260808 22:53:03.199174 14968 ts_tablet_manager.cc:1939] T 47e420296e6b4d23a5c882cee97d3078 P 3db1b2d6649143d4baca7f6702fab7a1: Deleting consensus metadata
I20260808 22:53:03.199410 14965 tablet_replica.cc:330] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: stopping tablet replica
I20260808 22:53:03.199549 14965 raft_consensus.cc:2229] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:53:03.199677 14965 raft_consensus.cc:2258] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:53:03.200253 14164 catalog_manager.cc:4841] TS 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): tablet 47e420296e6b4d23a5c882cee97d3078 (table test-table [id=56e17ee2fa474f57a844bf5b323002ac]) successfully deleted
I20260808 22:53:03.200536 14965 ts_tablet_manager.cc:1905] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:53:03.202121 14965 ts_tablet_manager.cc:1918] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:53:03.202191 14965 log.cc:1196] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/47e420296e6b4d23a5c882cee97d3078
I20260808 22:53:03.202452 14965 ts_tablet_manager.cc:1939] T 47e420296e6b4d23a5c882cee97d3078 P 8221083b44be4364b1f1d44273d0effb: Deleting consensus metadata
I20260808 22:53:03.202817 14163 catalog_manager.cc:4841] TS 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): tablet 47e420296e6b4d23a5c882cee97d3078 (table test-table [id=56e17ee2fa474f57a844bf5b323002ac]) successfully deleted
W20260808 22:53:03.206835 14162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:35677 (error 108)
I20260808 22:53:03.207029 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 14494
I20260808 22:53:03.208250 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:03.207175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35677 (local address 127.0.0.1:40852)
0808 22:53:03.208185 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:03.208222 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":993}
W20260808 22:53:03.208396 14162 catalog_manager.cc:4584] TS d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47e420296e6b4d23a5c882cee97d3078: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35677: connect: Connection refused (error 111)
I20260808 22:53:03.213711 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 14693
I20260808 22:53:03.220556 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 14133
I20260808 22:53:23.229481 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38615
--webserver_interface=127.22.30.62
--webserver_port=44115
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:23.308748 14972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:23.308936 14972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:23.308959 14972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:23.309990 14972 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 22:53:23.310019 14972 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 22:53:23.310032 14972 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 22:53:23.310043 14972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:53:23.310055 14972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:23.310067 14972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:23.310083 14972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:53:23.310098 14972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:53:23.311345 14972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:23.311576 14972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:23.311748 14972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:23.320689 14982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.320729 14984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:23.320749 14972 server_base.cc:1034] running on GCE node
W20260808 22:53:23.320708 14981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:23.321272 14972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:23.321612 14972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:23.322779 14972 hybrid_clock.cc:648] HybridClock initialized: now 1786229603322759 us; error 46 us; skew 500 ppm
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:23.325320 14972 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:53:23.325758 14972 webserver.cc:460] Webserver started at http://127.22.30.62:44115/ using document root <none> and password file <none>
I20260808 22:53:23.325985 14972 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:23.326035 14972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:23.327219 14972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:23.327821 14996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.328047 14972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.328127 14972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "c8d130379a7f4657ab2e0d3c9263b4b7"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ebc18c443f25ca0583182d26023064f2"
server_key_iv: "9f315fb0d9191214f3fbde14ece50993"
server_key_version: "encryptionkey@0"
I20260808 22:53:23.328480 14972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:23.335515 14972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:23.337447 14972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:23.337538 14972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:23.341225 14972 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:38615
I20260808 22:53:23.341293 15075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:38615 every 8 connection(s)
I20260808 22:53:23.341642 14972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:53:23.343017 15076 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:53:23.343668 15076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap starting.
I20260808 22:53:23.345880 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 14972
I20260808 22:53:23.346146 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:35677
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:38615
--webserver_port=38071
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:53:23.353430 15076 log.cc:824] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:23.355077 15076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:53:23.355356 15076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap complete.
I20260808 22:53:23.357600 15076 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:23.357825 15076 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d130379a7f4657ab2e0d3c9263b4b7, State: Initialized, Role: FOLLOWER
I20260808 22:53:23.357985 15076 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:23.358055 15076 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:53:23.358106 15076 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:53:23.358153 15076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 1 FOLLOWER]: Advancing to term 2
I20260808 22:53:23.359079 15076 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:23.359206 15076 leader_election.cc:302] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d130379a7f4657ab2e0d3c9263b4b7; no voters:
I20260808 22:53:23.359405 15076 leader_election.cc:288] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 22:53:23.359433 15083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 22:53:23.359537 15083 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 LEADER]: Becoming Leader. State: Replica: c8d130379a7f4657ab2e0d3c9263b4b7, State: Running, Role: LEADER
I20260808 22:53:23.359644 15076 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:53:23.359678 15083 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:23.360015 15086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } } }
I20260808 22:53:23.360016 15083 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d130379a7f4657ab2e0d3c9263b4b7. Latest consensus state: current_term: 2 leader_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } } }
I20260808 22:53:23.360133 15086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:53:23.360152 15083 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:53:23.360440 15097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:53:23.361196 15097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:53:23.361317 15097 catalog_manager.cc:1259] Loaded cluster ID: 3eca85c3dcc24cae9a7c0f6c98c10e78
I20260808 22:53:23.361364 15097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:53:23.362185 15097 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:53:23.362383 15097 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Loaded TSK: 0
I20260808 22:53:23.369757 15097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Generated new TSK 1
I20260808 22:53:23.369861 15097 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:53:23.441938 15081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:23.442143 15081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:23.442201 15081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:23.443315 15081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:23.443431 15081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:23.443499 15081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:53:23.444890 15081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:35677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=38071
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:23.445150 15081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:23.445365 15081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:23.449993 15121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.450033 15119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.450237 15124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:23.450350 15081 server_base.cc:1034] running on GCE node
I20260808 22:53:23.450515 15081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:23.450730 15081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:23.451882 15081 hybrid_clock.cc:648] HybridClock initialized: now 1786229603451870 us; error 29 us; skew 500 ppm
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:23.454305 15081 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:53:23.454681 15081 webserver.cc:460] Webserver started at http://127.22.30.1:38071/ using document root <none> and password file <none>
I20260808 22:53:23.454871 15081 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:23.454944 15081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:23.456231 15081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:23.456936 15134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.457098 15081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:23.457194 15081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "d843968709ce4395bad70fec8f2e5a8f"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ca6a34f859844dbcf58a66e079f11ac2"
server_key_iv: "878ff6788e643e5a14f99022d4648446"
server_key_version: "encryptionkey@0"
I20260808 22:53:23.457470 15081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:23.474278 15081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:23.476421 15081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:23.476567 15081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:23.476794 15081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:23.477187 15144 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:53:23.478916 15081 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:53:23.478973 15081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 22:53:23.479017 15081 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:53:23.479647 15081 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:53:23.479699 15081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:23.479765 15144 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap starting.
I20260808 22:53:23.487083 15081 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:35677
I20260808 22:53:23.487147 15327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:35677 every 8 connection(s)
I20260808 22:53:23.487536 15081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:53:23.490978 15144 log.cc:824] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:23.491896 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 15081
I20260808 22:53:23.492226 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:43235
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:38615
--webserver_port=32837
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:53:23.493578 15144 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:53:23.493836 15144 tablet_bootstrap.cc:492] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap complete.
I20260808 22:53:23.494978 15144 ts_tablet_manager.cc:1397] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:23.496163 15144 raft_consensus.cc:348] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.496424 15144 raft_consensus.cc:729] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843968709ce4395bad70fec8f2e5a8f, State: Initialized, Role: FOLLOWER
I20260808 22:53:23.496578 15144 consensus_queue.cc:260] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.496852 15144 ts_tablet_manager.cc:1428] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260808 22:53:23.499797 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.488202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:33481)
0808 22:53:23.488402 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:23.488448 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:23.489093 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:23.489579 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:23.489584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:23.489956 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:23.490229 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.490237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.491830 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.491831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.493053 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.493056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.493894 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.493895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.494015 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.494553 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:23.494562 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:23.496328 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:23.498211 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:23.498218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:23.498220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:23.498429 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:23.498671 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:23.498672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:23.498673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:23.498810 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:23.499187 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:23.499194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:23.499425 (+ 231us) client_negotiation.cc:769] Sending connection context
0808 22:53:23.499539 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 22:53:23.499671 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260808 22:53:23.500276 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.488467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:33481 (local address 127.22.30.62:38615)
0808 22:53:23.488658 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.488666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.488695 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:53:23.489309 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.489315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.489365 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.489518 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:23.490357 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.491685 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.493148 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.493711 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.493795 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.496458 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:23.496462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:23.496463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:23.496473 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:23.498090 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.498510 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.498513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.498514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.498601 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.498871 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.498872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.498872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.499074 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:23.499173 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.499991 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.500137 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260808 22:53:23.500883 15334 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38615
I20260808 22:53:23.500969 15334 heartbeater.cc:464] Registering TS with master...
I20260808 22:53:23.501134 15334 heartbeater.cc:511] Master 127.22.30.62:38615 requested a full tablet report, sending...
I20260808 22:53:23.501808 15018 ts_manager.cc:194] Registered new tserver with Master: d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677)
I20260808 22:53:23.502857 15018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:33481
I20260808 22:53:23.503863 15334 heartbeater.cc:503] Master 127.22.30.62:38615 was elected leader, sending a full tablet report...
I20260808 22:53:23.507197 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.502387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35677 (local address 127.0.0.1:55462)
0808 22:53:23.503498 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:23.503532 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:23.503593 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:23.503885 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:23.503888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:23.503890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:23.503945 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.503947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.504991 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.504992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.505864 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.505868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.506881 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.506882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.506979 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.506991 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:53:23.507054 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 22:53:23.507102 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":71,"threads_started":1}
I20260808 22:53:23.507624 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.502590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.22.30.1:35677)
0808 22:53:23.503710 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.503713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.503721 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:53:23.503754 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.503756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.503780 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.504158 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:23.504204 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.504904 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.505967 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.506714 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.506787 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.506842 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.507509 (+ 667us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.507549 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":64,"threads_started":1}
I20260808 22:53:23.509538 15214 tablet_service.cc:1514] Processing DeleteTablet for tablet 47e420296e6b4d23a5c882cee97d3078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:53:03 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:55462
I20260808 22:53:23.510049 15350 tablet_replica.cc:330] stopping tablet replica
I20260808 22:53:23.510156 15350 raft_consensus.cc:2229] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:53:23.510212 15350 raft_consensus.cc:2258] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:53:23.510527 15350 ts_tablet_manager.cc:1905] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:53:23.512688 15350 ts_tablet_manager.cc:1918] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:53:23.512789 15350 log.cc:1196] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal/wals/47e420296e6b4d23a5c882cee97d3078
I20260808 22:53:23.513051 15350 ts_tablet_manager.cc:1939] T 47e420296e6b4d23a5c882cee97d3078 P d843968709ce4395bad70fec8f2e5a8f: Deleting consensus metadata
I20260808 22:53:23.513813 14999 catalog_manager.cc:4841] TS d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677): tablet 47e420296e6b4d23a5c882cee97d3078 (table test-table [id=56e17ee2fa474f57a844bf5b323002ac]) successfully deleted
W20260808 22:53:23.576875 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:23.577039 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:23.577059 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:23.578126 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:23.578166 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:23.578189 15343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:53:23.579593 15343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:43235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=32837
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:23.579808 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:23.579972 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:23.584610 15358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.584622 15361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.584645 15355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:23.585036 15343 server_base.cc:1034] running on GCE node
I20260808 22:53:23.585213 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:23.585392 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:23.586526 15343 hybrid_clock.cc:648] HybridClock initialized: now 1786229603586514 us; error 28 us; skew 500 ppm
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:23.588967 15343 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:53:23.589315 15343 webserver.cc:460] Webserver started at http://127.22.30.2:32837/ using document root <none> and password file <none>
I20260808 22:53:23.589483 15343 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:23.589534 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:23.590764 15343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:23.591264 15377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.591439 15343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.591528 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "3db1b2d6649143d4baca7f6702fab7a1"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "db357d8cfbf016f3763199baf2623a2d"
server_key_iv: "8cae16332d578636fb439639fb6181e1"
server_key_version: "encryptionkey@0"
I20260808 22:53:23.591850 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:23.607241 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:23.609369 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:23.609521 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:23.609724 15343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:23.610018 15343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:23.610073 15343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.610109 15343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:23.610174 15343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.615926 15343 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:43235
I20260808 22:53:23.615968 15542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:43235 every 8 connection(s)
I20260808 22:53:23.616305 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:53:23.620287 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 15343
I20260808 22:53:23.620549 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:33591
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:38615
--webserver_port=46555
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:23.625195 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.616966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:53751 (local address 127.22.30.62:38615)
0808 22:53:23.617024 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.617026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.617215 (+ 189us) server_negotiation.cc:407] Connection header received
0808 22:53:23.617663 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.617664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.617676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.617742 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:23.618621 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.619100 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.620054 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.620249 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.620324 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.622415 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:23.622416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:23.622416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:23.622423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:23.623865 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.624208 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.624209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.624210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.624289 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.624518 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.624519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.624520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.624664 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:23.624730 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.624958 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.625054 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:53:23.625208 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.616941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:53751)
0808 22:53:23.617103 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:23.617139 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:23.617564 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:23.617812 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:23.617815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:23.618326 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:23.618522 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.618526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.619186 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.619187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.619964 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.619966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.620332 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.620333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.620463 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.620912 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:23.620936 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:23.622312 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:23.623941 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:23.623947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:23.623949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:23.624136 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:23.624364 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:23.624364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:23.624364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:23.624456 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:23.624725 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:23.624729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:23.624829 (+ 100us) client_negotiation.cc:769] Sending connection context
0808 22:53:23.624905 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 22:53:23.625007 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260808 22:53:23.625717 15544 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38615
I20260808 22:53:23.625823 15544 heartbeater.cc:464] Registering TS with master...
I20260808 22:53:23.625981 15544 heartbeater.cc:511] Master 127.22.30.62:38615 requested a full tablet report, sending...
I20260808 22:53:23.626399 15018 ts_manager.cc:194] Registered new tserver with Master: 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235)
I20260808 22:53:23.626920 15018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:53751
W20260808 22:53:23.700347 15548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:23.700505 15548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:23.700525 15548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:23.701501 15548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:23.701532 15548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:23.701556 15548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:53:23.702896 15548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=46555
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:23.703053 15548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:23.703204 15548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:23.707871 15555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.707837 15557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:23.707818 15554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:23.708088 15548 server_base.cc:1034] running on GCE node
I20260808 22:53:23.708312 15548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:23.708505 15548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:23.709647 15548 hybrid_clock.cc:648] HybridClock initialized: now 1786229603709632 us; error 29 us; skew 500 ppm
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:23.712112 15548 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:53:23.712540 15548 webserver.cc:460] Webserver started at http://127.22.30.3:46555/ using document root <none> and password file <none>
I20260808 22:53:23.712774 15548 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:23.712859 15548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:23.714111 15548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:53:23.714650 15572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.714819 15548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:53:23.714905 15548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "8221083b44be4364b1f1d44273d0effb"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "8a3dbc8925bcef58d17952ca43b9e971"
server_key_iv: "4989ccc0e57e692e869439871b1d4989"
server_key_version: "encryptionkey@0"
I20260808 22:53:23.715199 15548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:23.722369 15548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:23.724511 15548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:23.724659 15548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:23.724869 15548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:23.725180 15548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:23.725234 15548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.725279 15548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:23.725307 15548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:23.731482 15548 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:33591
I20260808 22:53:23.731559 15770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:33591 every 8 connection(s)
I20260808 22:53:23.731884 15548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:53:23.736720 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 15548
Aug 08 22:53:23 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229603, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:53:23.741322 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.732545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:51761 (local address 127.22.30.62:38615)
0808 22:53:23.732587 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.732589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.732815 (+ 226us) server_negotiation.cc:407] Connection header received
0808 22:53:23.733294 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.733296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.733309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.733352 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:23.734203 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.734679 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.735705 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.735910 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.736010 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.738507 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:23.738509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:23.738509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:23.738516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:23.739977 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.740373 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.740375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.740375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.740437 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.740682 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.740683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.740683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.740797 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:23.740868 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.741155 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.741239 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:53:23.741612 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.732539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:51761)
0808 22:53:23.732704 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:23.732735 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:23.733187 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:23.733432 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:23.733436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:23.733947 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:23.734113 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.734117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.734772 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.734773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.735614 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.735616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.736023 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.736024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.736141 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.736675 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:23.736682 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:23.738412 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:23.740078 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:23.740085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:23.740086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:23.740298 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:23.740503 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:23.740504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:23.740504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:23.740608 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:23.740873 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:23.740879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:23.741004 (+ 125us) client_negotiation.cc:769] Sending connection context
0808 22:53:23.741118 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 22:53:23.741262 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260808 22:53:23.742089 15771 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38615
I20260808 22:53:23.742179 15771 heartbeater.cc:464] Registering TS with master...
I20260808 22:53:23.742365 15771 heartbeater.cc:511] Master 127.22.30.62:38615 requested a full tablet report, sending...
I20260808 22:53:23.742796 15018 ts_manager.cc:194] Registered new tserver with Master: 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
I20260808 22:53:23.743170 15018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:51761
I20260808 22:53:23.744278 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.737195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.22.30.62:38615)
0808 22:53:23.737335 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.737337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.737421 (+ 84us) server_negotiation.cc:407] Connection header received
0808 22:53:23.737495 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.737496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.737506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.737556 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:23.738275 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.738724 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.739574 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.739781 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.739850 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.740994 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:23.740995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:23.740996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:23.741002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:23.743157 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.743612 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.743613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.743613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.743662 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.743867 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.743867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.743868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.743952 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:23.744068 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.744159 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.744216 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260808 22:53:23.745002 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:53:23.747618 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.745308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.22.30.62:38615)
0808 22:53:23.745365 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.745367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.745439 (+ 72us) server_negotiation.cc:407] Connection header received
0808 22:53:23.745526 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.745526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.745560 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.745633 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:53:23.745751 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.746171 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.746888 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.747088 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.747159 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.747331 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:53:23.747392 (+ 61us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 22:53:23.747547 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34272: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 22:53:23.747674 15776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34272: token expired
W20260808 22:53:23.747861 14913 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.30.62:38615)
I20260808 22:53:23.753064 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.748116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.22.30.62:38615)
0808 22:53:23.748289 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.748291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.748297 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:53:23.748314 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.748314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.748322 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.748369 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:23.748769 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.749163 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.749762 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.749921 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.749965 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.751131 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:23.751132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:23.751132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:23.751137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:23.752240 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.752524 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.752525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.752525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.752570 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:23.752713 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:23.752713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:23.752713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:23.752813 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:23.752878 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.752960 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.753015 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260808 22:53:23.754139 15018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:53:23.755636 15018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:53:23.757473 15214 tablet_service.cc:1467] Processing CreateTablet for tablet e7e15da66c154be784df6970acf8c727 (DEFAULT_TABLE table=test-table [id=503752a2d5bd42aeb1fa89c49ea8afea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:53:23.757682 15214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e15da66c154be784df6970acf8c727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:23.759076 15144 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap starting.
I20260808 22:53:23.759531 15144 tablet_bootstrap.cc:654] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:23.760174 15144 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: No bootstrap required, opened a new log
I20260808 22:53:23.760236 15144 ts_tablet_manager.cc:1397] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 22:53:23.760347 15144 raft_consensus.cc:348] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.760428 15144 raft_consensus.cc:374] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:23.760479 15144 raft_consensus.cc:729] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843968709ce4395bad70fec8f2e5a8f, State: Initialized, Role: FOLLOWER
I20260808 22:53:23.760538 15144 consensus_queue.cc:260] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.760648 15144 ts_tablet_manager.cc:1428] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 22:53:23.761365 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.757219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:33591 (local address 127.0.0.1:58282)
0808 22:53:23.757264 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:23.757284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:23.757322 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:23.757886 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:23.757888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:23.757890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:23.757926 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.757927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.759121 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.759122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.760285 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.760288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.761176 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.761177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.761230 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.761233 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:53:23.761262 (+ 29us) client_negotiation.cc:240] Negotiation successful
0808 22:53:23.761303 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 22:53:23.761519 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.757219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43235 (local address 127.0.0.1:54252)
0808 22:53:23.757600 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:23.757610 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:23.757667 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:23.758011 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:23.758012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:23.758014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:23.758061 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.758062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.759221 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.759222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.760358 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:23.760360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.761294 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.761295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:23.761345 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.761347 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:53:23.761402 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:53:23.761437 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":12,"thread_start_us":103,"threads_started":1}
I20260808 22:53:23.761583 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.757318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.22.30.2:43235)
0808 22:53:23.757481 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.757484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.757728 (+ 244us) server_negotiation.cc:407] Connection header received
0808 22:53:23.757820 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.757822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.757838 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.757913 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:23.758137 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.759144 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.760437 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.761142 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.761197 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.761252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.761443 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.761499 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260808 22:53:23.761801 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:23.757550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.22.30.3:33591)
0808 22:53:23.757725 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:23.757728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:23.757735 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:53:23.757785 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:23.757787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:23.757802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:23.757874 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:23.757976 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.759025 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.760411 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:23.761067 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:23.761526 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:23.761670 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:23.761698 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 22:53:23.761731 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260808 22:53:23.762212 15449 tablet_service.cc:1467] Processing CreateTablet for tablet e7e15da66c154be784df6970acf8c727 (DEFAULT_TABLE table=test-table [id=503752a2d5bd42aeb1fa89c49ea8afea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:53:23.762337 15659 tablet_service.cc:1467] Processing CreateTablet for tablet e7e15da66c154be784df6970acf8c727 (DEFAULT_TABLE table=test-table [id=503752a2d5bd42aeb1fa89c49ea8afea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:53:23.762550 15449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e15da66c154be784df6970acf8c727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:23.762686 15659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7e15da66c154be784df6970acf8c727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:53:23.765266 15787 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Bootstrap starting.
I20260808 22:53:23.765275 15789 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Bootstrap starting.
I20260808 22:53:23.765877 15789 tablet_bootstrap.cc:654] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:23.765962 15787 tablet_bootstrap.cc:654] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Neither blocks nor log segments found. Creating new log.
I20260808 22:53:23.766389 15789 log.cc:824] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:23.766430 15787 log.cc:824] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:23.767082 15789 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: No bootstrap required, opened a new log
I20260808 22:53:23.767098 15787 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: No bootstrap required, opened a new log
I20260808 22:53:23.767133 15789 ts_tablet_manager.cc:1397] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:53:23.767177 15787 ts_tablet_manager.cc:1397] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:53:23.768494 15789 raft_consensus.cc:348] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.768640 15789 raft_consensus.cc:374] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:23.768684 15789 raft_consensus.cc:729] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db1b2d6649143d4baca7f6702fab7a1, State: Initialized, Role: FOLLOWER
I20260808 22:53:23.768793 15789 consensus_queue.cc:260] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.768815 15787 raft_consensus.cc:348] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.768966 15787 raft_consensus.cc:374] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:53:23.769009 15787 raft_consensus.cc:729] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8221083b44be4364b1f1d44273d0effb, State: Initialized, Role: FOLLOWER
I20260808 22:53:23.769101 15789 ts_tablet_manager.cc:1428] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:53:23.769132 15544 heartbeater.cc:503] Master 127.22.30.62:38615 was elected leader, sending a full tablet report...
I20260808 22:53:23.769145 15787 consensus_queue.cc:260] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:23.769414 15787 ts_tablet_manager.cc:1428] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:53:23.769443 15771 heartbeater.cc:503] Master 127.22.30.62:38615 was elected leader, sending a full tablet report...
W20260808 22:53:23.866885 15545 tablet.cc:2369] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:53:23.982493 15772 tablet.cc:2369] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 22:53:23.989130 15339 tablet.cc:2369] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:53:24.038255 15799 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:24.038376 15799 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:24.038794 15799 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677)
I20260808 22:53:24.042382 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:24.039081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:53263 (local address 127.22.30.2:43235)
0808 22:53:24.039131 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:24.039134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:24.039171 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:53:24.039204 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:24.039205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:24.039218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:24.039275 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:24.039489 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.040038 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.041322 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.041986 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.042060 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:24.042070 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:24.042254 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 22:53:24.042305 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 22:53:24.042399 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:24.039107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:42333 (local address 127.22.30.1:35677)
0808 22:53:24.039225 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:24.039226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:24.039234 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:53:24.039329 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:24.039330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:24.039342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:24.039426 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:53:24.039607 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.040127 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.041343 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.041974 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.042044 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:24.042057 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:24.042278 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 22:53:24.042325 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260808 22:53:24.042411 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:24.038980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43235 (local address 127.22.30.3:53263)
0808 22:53:24.039051 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:24.039065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:24.039128 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:24.039294 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:24.039295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:24.039297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:24.039405 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:24.039407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.040116 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.040117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:24.041216 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:24.041220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.042105 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.042107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:24.042187 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:24.042191 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:53:24.042247 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 22:53:24.042307 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":20}
I20260808 22:53:24.042424 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:24.038999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35677 (local address 127.22.30.3:42333)
0808 22:53:24.039172 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:24.039183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:24.039232 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:24.039458 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:24.039459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:24.039462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:53:24.039536 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:24.039538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.040234 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.040235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:24.041229 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:24.041232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.042045 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.042046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:24.042180 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:24.042186 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:53:24.042251 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:53:24.042323 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":15,"thread_start_us":91,"threads_started":1}
I20260808 22:53:24.042829 15260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e15da66c154be784df6970acf8c727" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d843968709ce4395bad70fec8f2e5a8f" is_pre_election: true
I20260808 22:53:24.042874 15484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e15da66c154be784df6970acf8c727" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db1b2d6649143d4baca7f6702fab7a1" is_pre_election: true
I20260808 22:53:24.042982 15260 raft_consensus.cc:2454] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 0.
I20260808 22:53:24.043040 15484 raft_consensus.cc:2454] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 0.
I20260808 22:53:24.043218 15574 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8221083b44be4364b1f1d44273d0effb, d843968709ce4395bad70fec8f2e5a8f; no voters:
I20260808 22:53:24.043326 15799 raft_consensus.cc:2781] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:53:24.043437 15799 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:53:24.043470 15799 raft_consensus.cc:3037] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:24.044162 15799 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:24.044306 15799 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 election: Requested vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677)
I20260808 22:53:24.044478 15260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e15da66c154be784df6970acf8c727" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d843968709ce4395bad70fec8f2e5a8f"
I20260808 22:53:24.044478 15484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7e15da66c154be784df6970acf8c727" candidate_uuid: "8221083b44be4364b1f1d44273d0effb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db1b2d6649143d4baca7f6702fab7a1"
I20260808 22:53:24.044554 15260 raft_consensus.cc:3037] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:24.044554 15484 raft_consensus.cc:3037] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:53:24.045409 15260 raft_consensus.cc:2454] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 1.
I20260808 22:53:24.045409 15484 raft_consensus.cc:2454] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8221083b44be4364b1f1d44273d0effb in term 1.
I20260808 22:53:24.045614 15582 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb; no voters:
I20260808 22:53:24.045722 15799 raft_consensus.cc:2781] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:53:24.045864 15799 raft_consensus.cc:686] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 1 LEADER]: Becoming Leader. State: Replica: 8221083b44be4364b1f1d44273d0effb, State: Running, Role: LEADER
I20260808 22:53:24.045959 15799 consensus_queue.cc:237] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:24.046710 15018 catalog_manager.cc:5495] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8221083b44be4364b1f1d44273d0effb (127.22.30.3). New cstate: current_term: 1 leader_uuid: "8221083b44be4364b1f1d44273d0effb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } health_report { overall_health: UNKNOWN } } }
I20260808 22:53:24.114557 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:24.112417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.22.30.3:33591)
0808 22:53:24.112461 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:24.112463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:24.112471 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:53:24.112552 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:24.112553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:24.112571 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:24.112621 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:53:24.112742 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.113218 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.113881 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:24.114042 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:24.114106 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:24.114232 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:53:24.114339 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:53:24.114394 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:24.114426 (+ 32us) server_negotiation.cc:299] Negotiation successful
0808 22:53:24.114441 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:53:24.116394 15484 raft_consensus.cc:1261] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Refusing update from remote peer 8221083b44be4364b1f1d44273d0effb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:53:24.116393 15260 raft_consensus.cc:1261] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Refusing update from remote peer 8221083b44be4364b1f1d44273d0effb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:53:24.116753 15805 consensus_queue.cc:1035] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:53:24.116884 15799 consensus_queue.cc:1035] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:53:24.119066 15808 mvcc.cc:205] Tried to move back new op lower bound from 7316396458457784320 to 7316396458172780544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396458457784320}]
I20260808 22:53:24.119475 15810 mvcc.cc:205] Tried to move back new op lower bound from 7316396458457784320 to 7316396458172780544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396458457784320}]
I20260808 22:53:24.121618 15017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:53:24.121721 15017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:53:24.122460 15017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Sending DeleteTablet for 3 replicas of tablet e7e15da66c154be784df6970acf8c727
I20260808 22:53:24.122750 15659 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e15da66c154be784df6970acf8c727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:53:24 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:58282
I20260808 22:53:24.122792 15449 tablet_service.cc:1514] Processing DeleteTablet for tablet e7e15da66c154be784df6970acf8c727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:53:24 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:54252
I20260808 22:53:24.122818 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 15081
I20260808 22:53:24.126803 15819 tablet_replica.cc:330] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: stopping tablet replica
I20260808 22:53:24.126911 15819 raft_consensus.cc:2229] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:53:24.127067 15819 raft_consensus.cc:2258] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:53:24.127936 15819 ts_tablet_manager.cc:1905] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:53:24.128594 15822 tablet_replica.cc:330] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: stopping tablet replica
I20260808 22:53:24.128700 15822 raft_consensus.cc:2229] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:53:24.128790 15822 raft_consensus.cc:2258] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:53:24.130344 15822 ts_tablet_manager.cc:1905] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:53:24.130362 15819 ts_tablet_manager.cc:1918] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:53:24.130446 15819 log.cc:1196] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/e7e15da66c154be784df6970acf8c727
I20260808 22:53:24.130832 15819 ts_tablet_manager.cc:1939] T e7e15da66c154be784df6970acf8c727 P 8221083b44be4364b1f1d44273d0effb: Deleting consensus metadata
I20260808 22:53:24.132052 15822 ts_tablet_manager.cc:1918] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:53:24.132117 15822 log.cc:1196] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/e7e15da66c154be784df6970acf8c727
I20260808 22:53:24.132386 15822 ts_tablet_manager.cc:1939] T e7e15da66c154be784df6970acf8c727 P 3db1b2d6649143d4baca7f6702fab7a1: Deleting consensus metadata
I20260808 22:53:24.132762 15002 catalog_manager.cc:4841] TS 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): tablet e7e15da66c154be784df6970acf8c727 (table test-table [id=503752a2d5bd42aeb1fa89c49ea8afea]) successfully deleted
I20260808 22:53:24.137913 15001 catalog_manager.cc:4841] TS 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): tablet e7e15da66c154be784df6970acf8c727 (table test-table [id=503752a2d5bd42aeb1fa89c49ea8afea]) successfully deleted
W20260808 22:53:24.142810 14999 connection.cc:535] client connection to 127.22.30.1:35677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 22:53:24.142930 14999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:53:24.143241 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 15343
I20260808 22:53:24.143352 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:24.143196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35677 (local address 127.0.0.1:55478)
0808 22:53:24.143261 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:24.143315 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 22:53:24.148734 14999 catalog_manager.cc:4584] TS d843968709ce4395bad70fec8f2e5a8f (127.22.30.1:35677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7e15da66c154be784df6970acf8c727: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35677: connect: Connection refused (error 111)
I20260808 22:53:24.155700 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 15548
I20260808 22:53:24.164748 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 14972
I20260808 22:53:44.177024 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38615
--webserver_interface=127.22.30.62
--webserver_port=44115
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:44.254801 15827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:44.254982 15827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:44.255003 15827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:44.256071 15827 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 22:53:44.256102 15827 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 22:53:44.256114 15827 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 22:53:44.256125 15827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:53:44.256136 15827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:44.256148 15827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:44.256160 15827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:53:44.256194 15827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:53:44.257519 15827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=44115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:44.257682 15827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:44.257831 15827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:44.262892 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.262814 15835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.262814 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:44.262974 15827 server_base.cc:1034] running on GCE node
I20260808 22:53:44.263268 15827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:44.263576 15827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:44.264735 15827 hybrid_clock.cc:648] HybridClock initialized: now 1786229624264722 us; error 42 us; skew 500 ppm
Aug 08 22:53:44 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:44.267040 15827 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:53:44.267518 15827 webserver.cc:460] Webserver started at http://127.22.30.62:44115/ using document root <none> and password file <none>
I20260808 22:53:44.267728 15827 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:44.267807 15827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:44.269042 15827 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:44.269701 15854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.269879 15827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:53:44.269974 15827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "c8d130379a7f4657ab2e0d3c9263b4b7"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ebc18c443f25ca0583182d26023064f2"
server_key_iv: "9f315fb0d9191214f3fbde14ece50993"
server_key_version: "encryptionkey@0"
I20260808 22:53:44.270300 15827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:44.282895 15827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:44.284595 15827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:44.284750 15827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:44.288707 15827 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:38615
I20260808 22:53:44.288762 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:38615 every 8 connection(s)
I20260808 22:53:44.289114 15827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:53:44.290665 15937 sys_catalog.cc:263] Verifying existing consensus state
I20260808 22:53:44.291275 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap starting.
I20260808 22:53:44.292842 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 15827
I20260808 22:53:44.293109 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:35677
--local_ip_for_outbound_sockets=127.22.30.1
--tserver_master_addrs=127.22.30.62:38615
--webserver_port=38071
--webserver_interface=127.22.30.1
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:53:44.300455 15937 log.cc:824] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:44.302662 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:53:44.302958 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Bootstrap complete.
I20260808 22:53:44.305449 15937 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:44.305661 15937 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d130379a7f4657ab2e0d3c9263b4b7, State: Initialized, Role: FOLLOWER
I20260808 22:53:44.305825 15937 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:44.305893 15937 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:53:44.305941 15937 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:53:44.305986 15937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 2 FOLLOWER]: Advancing to term 3
I20260808 22:53:44.306922 15937 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:44.307034 15937 leader_election.cc:302] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d130379a7f4657ab2e0d3c9263b4b7; no voters:
I20260808 22:53:44.307190 15937 leader_election.cc:288] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 22:53:44.307250 15944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 22:53:44.307408 15944 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [term 3 LEADER]: Becoming Leader. State: Replica: c8d130379a7f4657ab2e0d3c9263b4b7, State: Running, Role: LEADER
I20260808 22:53:44.307473 15937 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:53:44.307502 15944 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } }
I20260808 22:53:44.307770 15944 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d130379a7f4657ab2e0d3c9263b4b7. Latest consensus state: current_term: 3 leader_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } } }
I20260808 22:53:44.307842 15944 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:53:44.307976 15945 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d130379a7f4657ab2e0d3c9263b4b7" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38615 } } }
I20260808 22:53:44.308025 15945 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7 [sys.catalog]: This master's current role is: LEADER
I20260808 22:53:44.308261 15951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:53:44.309053 15951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:53:44.309151 15951 catalog_manager.cc:1259] Loaded cluster ID: 3eca85c3dcc24cae9a7c0f6c98c10e78
I20260808 22:53:44.309182 15951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:53:44.310417 15951 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:53:44.310590 15951 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Loaded TSK: 0
I20260808 22:53:44.310631 15951 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8d130379a7f4657ab2e0d3c9263b4b7: Loaded TSK: 1
F20260808 22:53:44.310837 15951 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9c744e9ad google::LogMessage::Fail() at ??:0
@ 0x7fb9c745188a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9c744e4de google::LogMessage::Flush() at ??:0
@ 0x7fb9c744fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9c9bbb104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb9c9bb8d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb9cf3a81cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb9cf3a848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb9cf3a8594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb9cf36ec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb9cf378ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9c931c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9c9311954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9ca1756db start_thread at ??:0
@ 0x7fb9c5f4c61f clone at ??:0
W20260808 22:53:44.388614 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:44.388870 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:44.388906 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:44.389966 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:44.390002 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:44.390026 15942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:53:44.391348 15942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:35677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=38071
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:44.391621 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:44.391834 15942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:44.396438 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.396413 16003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.396642 16004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:44.396720 15942 server_base.cc:1034] running on GCE node
I20260808 22:53:44.396880 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:44.397125 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:44.398277 15942 hybrid_clock.cc:648] HybridClock initialized: now 1786229624398255 us; error 34 us; skew 500 ppm
Aug 08 22:53:44 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:44.400732 15942 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:53:44.401176 15942 webserver.cc:460] Webserver started at http://127.22.30.1:38071/ using document root <none> and password file <none>
I20260808 22:53:44.401392 15942 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:44.401458 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:44.402700 15942 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:44.403389 16020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.403589 15942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:44.403685 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "d843968709ce4395bad70fec8f2e5a8f"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "ca6a34f859844dbcf58a66e079f11ac2"
server_key_iv: "878ff6788e643e5a14f99022d4648446"
server_key_version: "encryptionkey@0"
I20260808 22:53:44.404120 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:44.414551 15942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:44.416816 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:44.416988 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:44.417213 15942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:44.417712 16035 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 22:53:44.419152 15942 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 22:53:44.419211 15942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 22:53:44.419241 15942 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 22:53:44.420097 15942 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 22:53:44.420143 15942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 22:53:44.420239 16035 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap starting.
I20260808 22:53:44.427417 15942 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:35677
I20260808 22:53:44.427498 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:35677 every 8 connection(s)
I20260808 22:53:44.427824 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:53:44.428690 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 15942
I20260808 22:53:44.429024 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:43235
--local_ip_for_outbound_sockets=127.22.30.2
--tserver_master_addrs=127.22.30.62:38615
--webserver_port=32837
--webserver_interface=127.22.30.2
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:53:44.430721 16035 log.cc:824] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Log is configured to *not* fsync() on all Append() calls
I20260808 22:53:44.433329 16035 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 22:53:44.433633 16035 tablet_bootstrap.cc:492] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Bootstrap complete.
I20260808 22:53:44.434974 16035 ts_tablet_manager.cc:1397] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260808 22:53:44.436600 16035 raft_consensus.cc:348] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:44.436991 16035 raft_consensus.cc:729] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843968709ce4395bad70fec8f2e5a8f, State: Initialized, Role: FOLLOWER
I20260808 22:53:44.437161 16035 consensus_queue.cc:260] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:44.437463 16035 ts_tablet_manager.cc:1428] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260808 22:53:44.511202 16224 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:53:44.428503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:34593)
0808 22:53:44.431281 (+ 2778us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.431323 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:44.431894 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:44.511052 (+ 79158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2728,"thread_start_us":92,"threads_started":1}
W20260808 22:53:44.511489 16219 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:38615 (0 consecutive failures): Network error: Failed to ping master at 127.22.30.62:38615: Client connection negotiation failed: client connection to 127.22.30.62:38615: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 22:53:44.511827 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.511699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:57633)
0808 22:53:44.511751 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.511803 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 22:53:44.512212 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.512097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:52135)
0808 22:53:44.512150 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.512183 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 22:53:44.512387 16219 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 22:53:44.513707 16225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:44.513868 16225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:44.513918 16225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:44.515017 16225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:44.515054 16225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:44.515103 16225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:53:44.516610 16225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:43235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=32837
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:44.516842 16225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:44.517064 16225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:44.522051 16235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.522060 16239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.522114 16234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:44.522286 16225 server_base.cc:1034] running on GCE node
I20260808 22:53:44.522509 16225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:44.522751 16225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:44.523919 16225 hybrid_clock.cc:648] HybridClock initialized: now 1786229624523913 us; error 37 us; skew 500 ppm
Aug 08 22:53:44 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:44.526257 16225 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:53:44.526650 16225 webserver.cc:460] Webserver started at http://127.22.30.2:32837/ using document root <none> and password file <none>
I20260808 22:53:44.526875 16225 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:44.526942 16225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:44.528245 16225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:53:44.528824 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.528981 16225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:53:44.529076 16225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "3db1b2d6649143d4baca7f6702fab7a1"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "db357d8cfbf016f3763199baf2623a2d"
server_key_iv: "8cae16332d578636fb439639fb6181e1"
server_key_version: "encryptionkey@0"
I20260808 22:53:44.529399 16225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:44.540735 16225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:44.542377 16225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:44.542522 16225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:44.542748 16225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:44.543033 16225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:44.543085 16225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.543123 16225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:44.543167 16225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.549086 16225 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:43235
I20260808 22:53:44.549180 16429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:43235 every 8 connection(s)
I20260808 22:53:44.549479 16225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:53:44.550464 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.550136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:52757)
0808 22:53:44.550274 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.550335 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260808 22:53:44.550865 16430 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:38615 (0 consecutive failures): Network error: Failed to ping master at 127.22.30.62:38615: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
I20260808 22:53:44.551144 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.551060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:38969)
0808 22:53:44.551103 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.551127 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 22:53:44.551409 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.551319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:47111)
0808 22:53:44.551355 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.551394 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 22:53:44.551505 16430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:53:44.556342 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 16225
I20260808 22:53:44.556602 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:33591
--local_ip_for_outbound_sockets=127.22.30.3
--tserver_master_addrs=127.22.30.62:38615
--webserver_port=46555
--webserver_interface=127.22.30.3
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:53:44.634673 16433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:53:44.634850 16433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:53:44.634872 16433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:53:44.635928 16433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:53:44.635962 16433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:53:44.635985 16433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:53:44.637270 16433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=46555
--tserver_master_addrs=127.22.30.62:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:53:44.637444 16433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:53:44.637601 16433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:53:44.642436 16438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:53:44.642436 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:44.642452 16433 server_base.cc:1034] running on GCE node
W20260808 22:53:44.642449 16439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:53:44.642890 16433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:53:44.643065 16433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:53:44.644200 16433 hybrid_clock.cc:648] HybridClock initialized: now 1786229624644186 us; error 30 us; skew 500 ppm
Aug 08 22:53:44 dist-test-slave-x4qh krb5kdc[14109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:53:44.646697 16433 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:53:44.647082 16433 webserver.cc:460] Webserver started at http://127.22.30.3:46555/ using document root <none> and password file <none>
I20260808 22:53:44.647320 16433 fs_manager.cc:359] Metadata directory not provided
I20260808 22:53:44.647403 16433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:53:44.648603 16433 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:53:44.649140 16460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.649298 16433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:53:44.649428 16433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "8221083b44be4364b1f1d44273d0effb"
format_stamp: "Formatted at 2026-08-08 22:53:02 on dist-test-slave-x4qh"
server_key: "8a3dbc8925bcef58d17952ca43b9e971"
server_key_iv: "4989ccc0e57e692e869439871b1d4989"
server_key_version: "encryptionkey@0"
I20260808 22:53:44.649698 16433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:53:44.659475 16433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:53:44.661048 16433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:53:44.661201 16433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:53:44.661401 16433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:53:44.661691 16433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:53:44.661738 16433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.661767 16433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:53:44.661784 16433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:53:44.667654 16433 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:33591
I20260808 22:53:44.667740 16633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:33591 every 8 connection(s)
I20260808 22:53:44.668004 16433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:53:44.668903 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.668577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:60987)
0808 22:53:44.668719 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.668780 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260808 22:53:44.669255 16634 heartbeater.cc:644] Failed to heartbeat to 127.22.30.62:38615 (0 consecutive failures): Network error: Failed to ping master at 127.22.30.62:38615: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
I20260808 22:53:44.669525 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.669440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:46903)
0808 22:53:44.669484 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.669509 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 22:53:44.669807 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:44.669736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:46013)
0808 22:53:44.669778 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:44.669795 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:53:44.669927 16634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 22:53:44.671929 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 16433
I20260808 22:53:45.512933 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:45.512688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:55139)
0808 22:53:45.512801 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:45.512859 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260808 22:53:45.551967 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:45.551783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:55525)
0808 22:53:45.551889 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:45.551918 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260808 22:53:45.670411 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:45.670209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:53989)
0808 22:53:45.670312 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:45.670363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20260808 22:53:46.031109 16645 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:46.031244 16645 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:46.031603 16645 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
Aug 08 22:53:46 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.3@KRBTEST.COM
Aug 08 22:53:46 dist-test-slave-x4qh krb5kdc[14109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.2@KRBTEST.COM
I20260808 22:53:46.046167 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.031748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:33591 (local address 127.22.30.1:53973)
0808 22:53:46.031895 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:46.031905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:46.031959 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:46.032952 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:46.032955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:46.033453 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:46.033709 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:46.033716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.035300 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.035301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:46.036521 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:46.036524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.037337 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.037337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:46.037469 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:46.040867 (+ 3398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:46.040877 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:46.042484 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:46.044593 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:46.044601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:46.044603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:46.044912 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:46.045178 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:46.045179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:46.045179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:46.045308 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:46.045691 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:46.045698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:46.045822 (+ 124us) client_negotiation.cc:769] Sending connection context
0808 22:53:46.045922 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 22:53:46.046087 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":49,"thread_start_us":82,"threads_started":1}
I20260808 22:53:46.046177 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.031876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:39923 (local address 127.22.30.2:43235)
0808 22:53:46.032030 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:46.032041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:46.032075 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:53:46.032622 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:46.032626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:46.032680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:46.032869 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:46.033828 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.035181 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.036603 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.037166 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.037267 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:46.043059 (+ 5792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:46.043063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:46.043063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:46.043071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:46.044685 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:46.045099 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:46.045101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:46.045102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:46.045218 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:46.045445 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:46.045445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:46.045446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:46.045635 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:46.045726 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:46.045976 (+ 250us) server_negotiation.cc:299] Negotiation successful
0808 22:53:46.046075 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":45,"threads_started":1}
I20260808 22:53:46.046415 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.031862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:53973 (local address 127.22.30.3:33591)
0808 22:53:46.032030 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 22:53:46.032041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 22:53:46.032075 (+ 34us) server_negotiation.cc:407] Connection header received
0808 22:53:46.032623 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:53:46.032626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:53:46.032680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:53:46.032915 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:53:46.033828 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.035171 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.036604 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.037166 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.037275 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:46.042585 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:53:46.042589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:53:46.042589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:53:46.042600 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:53:46.044496 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:46.044995 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:46.044998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:46.044998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:46.045109 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:53:46.045386 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:53:46.045386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:53:46.045387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:53:46.045622 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:53:46.045678 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:53:46.045985 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 22:53:46.046116 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":50,"threads_started":1}
I20260808 22:53:46.046612 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.031748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:43235 (local address 127.22.30.1:39923)
0808 22:53:46.031895 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:46.031905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:53:46.031959 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:53:46.032952 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:53:46.032955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:53:46.033453 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:53:46.033709 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:46.033716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.035297 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.035298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:46.036521 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:53:46.036524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:53:46.037335 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:53:46.037336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:53:46.037469 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:53:46.040867 (+ 3398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:53:46.040877 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:53:46.042953 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:53:46.044768 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:46.044769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:46.044769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:46.045020 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:46.045296 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:53:46.045296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:53:46.045297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:53:46.045383 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:53:46.045725 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:53:46.045725 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:53:46.045815 (+ 90us) client_negotiation.cc:769] Sending connection context
0808 22:53:46.045924 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 22:53:46.046158 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":196,"thread_start_us":87,"threads_started":1}
W20260808 22:53:46.047070 16029 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
W20260808 22:53:46.047277 16030 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
I20260808 22:53:46.047322 16030 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d843968709ce4395bad70fec8f2e5a8f; no voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb
I20260808 22:53:46.047474 16645 raft_consensus.cc:2726] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:46.513593 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.513419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:51327)
0808 22:53:46.513518 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:46.513554 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 22:53:46.552629 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.552386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:52309)
0808 22:53:46.552543 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:46.552580 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260808 22:53:46.671092 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:46.670894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:59661)
0808 22:53:46.671024 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:46.671056 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
I20260808 22:53:47.514230 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:47.514010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:45035)
0808 22:53:47.514124 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:47.514157 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260808 22:53:47.553150 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:47.552978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:46405)
0808 22:53:47.553069 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:47.553096 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260808 22:53:47.556413 16661 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:47.556485 16661 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:47.556641 16661 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
W20260808 22:53:47.557070 16030 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
W20260808 22:53:47.557134 16029 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
I20260808 22:53:47.557201 16029 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d843968709ce4395bad70fec8f2e5a8f; no voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb
I20260808 22:53:47.557297 16661 raft_consensus.cc:2726] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:47.671993 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:47.671663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:37499)
0808 22:53:47.671908 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:47.671944 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260808 22:53:48.514919 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:48.514699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:50959)
0808 22:53:48.514839 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:48.514874 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
I20260808 22:53:48.553732 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:48.553557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:48975)
0808 22:53:48.553663 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:48.553700 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260808 22:53:48.672598 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:48.672417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:36675)
0808 22:53:48.672537 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:48.672567 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260808 22:53:49.515632 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:49.515420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:39867)
0808 22:53:49.515548 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:49.515586 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260808 22:53:49.554450 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:49.554238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:39663)
0808 22:53:49.554377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:49.554411 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260808 22:53:49.673170 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:49.672975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:50297)
0808 22:53:49.673102 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:49.673134 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260808 22:53:49.885840 16673 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:49.885952 16673 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:49.886150 16673 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
W20260808 22:53:49.886511 16029 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
W20260808 22:53:49.886637 16030 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
I20260808 22:53:49.886706 16030 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d843968709ce4395bad70fec8f2e5a8f; no voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb
I20260808 22:53:49.886770 16673 raft_consensus.cc:2726] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:50.516275 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:50.516067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:56501)
0808 22:53:50.516202 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:50.516230 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260808 22:53:50.555071 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:50.554856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:37849)
0808 22:53:50.554981 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:50.555034 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260808 22:53:50.673790 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:50.673576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:51853)
0808 22:53:50.673693 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:50.673744 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260808 22:53:51.516904 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:51.516683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:43485)
0808 22:53:51.516797 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:51.516826 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260808 22:53:51.555730 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:51.555529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:60693)
0808 22:53:51.555658 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:51.555690 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260808 22:53:51.674345 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:51.674163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:49337)
0808 22:53:51.674276 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:51.674310 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260808 22:53:52.517493 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:52.517293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:59027)
0808 22:53:52.517420 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:52.517450 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260808 22:53:52.522456 16683 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:52.522542 16683 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:52.522769 16683 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
W20260808 22:53:52.523099 16030 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
W20260808 22:53:52.523175 16029 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
I20260808 22:53:52.523254 16029 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d843968709ce4395bad70fec8f2e5a8f; no voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb
I20260808 22:53:52.523320 16683 raft_consensus.cc:2726] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:52.556330 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:52.556130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:51479)
0808 22:53:52.556272 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:52.556300 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260808 22:53:52.674991 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:52.674769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:57377)
0808 22:53:52.674913 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:52.674940 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260808 22:53:53.518265 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:53.518081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:49225)
0808 22:53:53.518191 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:53.518220 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260808 22:53:53.556886 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:53.556700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:43205)
0808 22:53:53.556817 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:53.556856 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260808 22:53:53.675661 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:53.675489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:37931)
0808 22:53:53.675597 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:53.675626 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260808 22:53:54.518930 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:54.518724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:43111)
0808 22:53:54.518853 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:54.518885 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260808 22:53:54.557463 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:54.557257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:35427)
0808 22:53:54.557392 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:54.557426 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260808 22:53:54.676285 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:54.676074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:36951)
0808 22:53:54.676199 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:54.676247 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260808 22:53:55.519613 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:55.519400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:39297)
0808 22:53:55.519530 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:55.519568 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260808 22:53:55.558185 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:55.557917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:52605)
0808 22:53:55.558114 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:55.558147 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":53,"threads_started":1}
I20260808 22:53:55.676867 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:55.676658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:37927)
0808 22:53:55.676797 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:55.676825 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260808 22:53:56.520233 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:56.519998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:42779)
0808 22:53:56.520158 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:56.520193 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260808 22:53:56.558876 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:56.558673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:59457)
0808 22:53:56.558804 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:56.558840 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260808 22:53:56.677418 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:56.677236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:59885)
0808 22:53:56.677337 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:56.677366 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260808 22:53:57.516006 16708 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:57.516089 16708 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:57.516217 16708 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
W20260808 22:53:57.516578 16029 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
W20260808 22:53:57.516695 16030 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
I20260808 22:53:57.516729 16030 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d843968709ce4395bad70fec8f2e5a8f; no voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb
I20260808 22:53:57.516816 16708 raft_consensus.cc:2726] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:57.520862 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:57.520690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:54077)
0808 22:53:57.520798 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:57.520826 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260808 22:53:57.559552 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:57.559313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:34025)
0808 22:53:57.559487 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:57.559519 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260808 22:53:57.678103 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:57.677862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:35831)
0808 22:53:57.678023 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:57.678054 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260808 22:53:58.521553 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:58.521341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:53653)
0808 22:53:58.521479 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:58.521510 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260808 22:53:58.560149 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:58.559944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:47485)
0808 22:53:58.560083 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:58.560114 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260808 22:53:58.678668 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:58.678466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:58615)
0808 22:53:58.678596 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:58.678627 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260808 22:53:59.130939 16720 raft_consensus.cc:482] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:53:59.131045 16720 raft_consensus.cc:504] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db1b2d6649143d4baca7f6702fab7a1" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 43235 } } peers { permanent_uuid: "8221083b44be4364b1f1d44273d0effb" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33591 } } peers { permanent_uuid: "d843968709ce4395bad70fec8f2e5a8f" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35677 } }
I20260808 22:53:59.131266 16720 leader_election.cc:288] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235), 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591)
W20260808 22:53:59.131690 16029 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8221083b44be4364b1f1d44273d0effb (127.22.30.3:33591): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
W20260808 22:53:59.131788 16030 leader_election.cc:341] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3db1b2d6649143d4baca7f6702fab7a1 (127.22.30.2:43235): Not found: Tablet not found: e7e15da66c154be784df6970acf8c727
I20260808 22:53:59.131865 16030 leader_election.cc:302] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d843968709ce4395bad70fec8f2e5a8f; no voters: 3db1b2d6649143d4baca7f6702fab7a1, 8221083b44be4364b1f1d44273d0effb
I20260808 22:53:59.131938 16720 raft_consensus.cc:2726] T e7e15da66c154be784df6970acf8c727 P d843968709ce4395bad70fec8f2e5a8f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 22:53:59.522347 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:59.522089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.1:42599)
0808 22:53:59.522249 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:59.522304 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260808 22:53:59.560798 16723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:59.560560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.2:58727)
0808 22:53:59.560708 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:59.560742 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260808 22:53:59.675638 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 15942
I20260808 22:53:59.679824 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:53:59.679620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38615 (local address 127.22.30.3:49909)
0808 22:53:59.679729 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 22:53:59.679760 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.62:38615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260808 22:53:59.687038 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 16225
I20260808 22:53:59.696187 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 16433
2026-08-08T22:53:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58598 ms)
[----------] 30 tests from SecurityITest (229122 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:53:59 dist-test-slave-x4qh krb5kdc[16734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:53:59 dist-test-slave-x4qh krb5kdc[16734](info): set up 2 sockets
Aug 08 22:53:59 dist-test-slave-x4qh krb5kdc[16734](info): commencing operation
krb5kdc: starting...
W20260808 22:54:00.889106 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:00 dist-test-slave-x4qh krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:01.000240 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38631
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:38631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:01.074441 16753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:01.074617 16753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:01.074638 16753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:01.075668 16753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:01.075702 16753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:01.075716 16753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:01.075728 16753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:01.075740 16753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:01.077126 16753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:38631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:38631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:01.077288 16753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:01.077427 16753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:54:01.082149 16753 server_base.cc:1034] running on GCE node
W20260808 22:54:01.082165 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.082194 16762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.082305 16764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.082603 16753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:01.082854 16753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:01.083981 16753 hybrid_clock.cc:648] HybridClock initialized: now 1786229641083966 us; error 34 us; skew 500 ppm
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:01.086186 16753 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:01.086598 16753 webserver.cc:460] Webserver started at http://127.22.30.62:41529/ using document root <none> and password file <none>
I20260808 22:54:01.086807 16753 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:01.086869 16753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:01.087004 16753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:01.088032 16753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "ea37340cdb484e63ace39d43e023976b"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "edf5328713bd3d974fff010d4f36f398"
server_key_iv: "be4b18c58736c4ac9263a04fad1c1418"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.088389 16753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "ea37340cdb484e63ace39d43e023976b"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "edf5328713bd3d974fff010d4f36f398"
server_key_iv: "be4b18c58736c4ac9263a04fad1c1418"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.089617 16753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:54:01.090237 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.090394 16753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:01.090481 16753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "ea37340cdb484e63ace39d43e023976b"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "edf5328713bd3d974fff010d4f36f398"
server_key_iv: "be4b18c58736c4ac9263a04fad1c1418"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.090549 16753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:01.100656 16753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:01.102119 16753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:01.102258 16753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:01.106366 16753 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:38631
I20260808 22:54:01.106434 16863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:38631 every 8 connection(s)
I20260808 22:54:01.106870 16753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:01.107672 16871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:01.109510 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b: Bootstrap starting.
I20260808 22:54:01.110028 16871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:01.110291 16871 log.cc:824] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:01.110901 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b: No bootstrap required, opened a new log
I20260808 22:54:01.112304 16871 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea37340cdb484e63ace39d43e023976b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38631 } }
I20260808 22:54:01.112437 16871 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:01.112480 16871 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea37340cdb484e63ace39d43e023976b, State: Initialized, Role: FOLLOWER
I20260808 22:54:01.112603 16871 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea37340cdb484e63ace39d43e023976b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38631 } }
I20260808 22:54:01.112672 16871 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:01.112715 16871 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:01.112759 16871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:01.113222 16871 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea37340cdb484e63ace39d43e023976b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38631 } }
I20260808 22:54:01.113324 16871 leader_election.cc:302] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea37340cdb484e63ace39d43e023976b; no voters:
I20260808 22:54:01.113477 16871 leader_election.cc:288] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:01.113503 16877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:01.113665 16877 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [term 1 LEADER]: Becoming Leader. State: Replica: ea37340cdb484e63ace39d43e023976b, State: Running, Role: LEADER
I20260808 22:54:01.113692 16871 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:01.113739 16877 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea37340cdb484e63ace39d43e023976b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38631 } }
I20260808 22:54:01.114108 16880 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea37340cdb484e63ace39d43e023976b. Latest consensus state: current_term: 1 leader_uuid: "ea37340cdb484e63ace39d43e023976b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea37340cdb484e63ace39d43e023976b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38631 } } }
I20260808 22:54:01.114182 16880 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:01.114355 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 16753
I20260808 22:54:01.114461 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:01.114523 16889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:01.114683 22648 external_mini_cluster.cc:1428] Setting key c7df18ad399717bd65d52b27651cd9b2
I20260808 22:54:01.114980 16889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:01.115339 16878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea37340cdb484e63ace39d43e023976b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea37340cdb484e63ace39d43e023976b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 38631 } } }
I20260808 22:54:01.115475 16878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:01.116490 16889 catalog_manager.cc:1347] Generated new cluster ID: 0d186865afe64e6cb1cb5ad80508ab06
I20260808 22:54:01.116544 16889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:01.122992 16889 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:01.123574 16889 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:01.124054 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.115109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.22.30.62:38631)
0808 22:54:01.115246 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.115253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.115665 (+ 412us) server_negotiation.cc:407] Connection header received
0808 22:54:01.115752 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.115757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.115797 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.115923 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:01.116657 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.117574 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.118320 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.118688 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.118790 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.120922 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:01.120926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:01.120926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:01.120936 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:01.122385 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.122879 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.122882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.122882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.122981 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.123189 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.123190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.123190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.123474 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:01.123622 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.123732 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.123871 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260808 22:54:01.133091 16889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b: Generated new TSK 0
I20260808 22:54:01.133235 16889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:54:01.157759 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38631
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:01.232705 16910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:01.232887 16910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:01.232908 16910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:01.233879 16910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:01.233911 16910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:01.233934 16910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:54:01.235163 16910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:01.235327 16910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:01.235553 16910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:54:01.240131 16910 server_base.cc:1034] running on GCE node
W20260808 22:54:01.240067 16919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.240088 16917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.240072 16921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.240517 16910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:01.240760 16910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:01.241910 16910 hybrid_clock.cc:648] HybridClock initialized: now 1786229641241895 us; error 28 us; skew 500 ppm
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:01.244189 16910 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:54:01.244589 16910 webserver.cc:460] Webserver started at http://127.22.30.1:40651/ using document root <none> and password file <none>
I20260808 22:54:01.244798 16910 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:01.244861 16910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:01.244998 16910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:01.245972 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "fa8834ae230745e9afea58390cc359c0"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "89fac25c7c91f5515a273afc72ef2074"
server_key_iv: "1fd743663b55fb21ad5b72e3a5474b4a"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.246328 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "fa8834ae230745e9afea58390cc359c0"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "89fac25c7c91f5515a273afc72ef2074"
server_key_iv: "1fd743663b55fb21ad5b72e3a5474b4a"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.247609 16910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:01.248325 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.248485 16910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:01.248574 16910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "fa8834ae230745e9afea58390cc359c0"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "89fac25c7c91f5515a273afc72ef2074"
server_key_iv: "1fd743663b55fb21ad5b72e3a5474b4a"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.248663 16910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:01.265296 16910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:01.266883 16910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:01.267019 16910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:01.267212 16910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:01.267539 16910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:01.267588 16910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.267619 16910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:01.267645 16910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.272903 16910 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:34151
I20260808 22:54:01.272980 17100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:34151 every 8 connection(s)
I20260808 22:54:01.273269 16910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:01.281399 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.273915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:43269 (local address 127.22.30.62:38631)
0808 22:54:01.273981 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.273983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.274164 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:54:01.274584 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.274585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.274593 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.274643 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:01.275465 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.275957 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.277005 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.277307 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.277373 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.279205 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:01.279207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:01.279207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:01.279213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:01.280372 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.280702 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.280703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.280703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.280751 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.280939 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.280940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.280940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.281030 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:01.281075 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.281268 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.281345 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 22:54:01.281509 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.273882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38631 (local address 127.22.30.1:43269)
0808 22:54:01.274032 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.274082 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.274499 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.274689 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.274691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.275176 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:01.275333 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.275337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.276033 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.276034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.276825 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.276827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.277391 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.277392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.277479 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.277835 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:01.277841 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:01.279124 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:01.280436 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:01.280442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:01.280443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:01.280632 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:01.280802 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:01.280803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:01.280803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:01.280891 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:01.281078 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:01.281082 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:01.281174 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.281245 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.281360 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260808 22:54:01.281975 17101 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38631
I20260808 22:54:01.281980 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 16910
I20260808 22:54:01.282066 17101 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:01.282085 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:54:01.282232 17101 heartbeater.cc:511] Master 127.22.30.62:38631 requested a full tablet report, sending...
I20260808 22:54:01.282299 22648 external_mini_cluster.cc:1428] Setting key a3d0e87656bbdf7b700d10d658c50a5e
I20260808 22:54:01.282740 16802 ts_manager.cc:194] Registered new tserver with Master: fa8834ae230745e9afea58390cc359c0 (127.22.30.1:34151)
I20260808 22:54:01.283358 16802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:43269
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:54:01.305935 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38631
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:01.382541 17107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:01.382730 17107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:01.382766 17107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:01.383803 17107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:01.383857 17107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:01.383895 17107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:54:01.385205 17107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:01.385442 17107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:01.385634 17107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:01.390055 17118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.390061 17116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.390157 17107 server_base.cc:1034] running on GCE node
W20260808 22:54:01.390296 17115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.390475 17107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:01.390720 17107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:01.391863 17107 hybrid_clock.cc:648] HybridClock initialized: now 1786229641391859 us; error 37 us; skew 500 ppm
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:01.394258 17107 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:54:01.394623 17107 webserver.cc:460] Webserver started at http://127.22.30.2:41993/ using document root <none> and password file <none>
I20260808 22:54:01.394809 17107 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:01.394848 17107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:01.394971 17107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:01.395953 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "0059864cab15436b886044511cb5004a"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "e48b3088ac15025ae649a3776927eda9"
server_key_iv: "776936030791529be983b58c8f92b10d"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.396307 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "0059864cab15436b886044511cb5004a"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "e48b3088ac15025ae649a3776927eda9"
server_key_iv: "776936030791529be983b58c8f92b10d"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.397606 17107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:01.398221 17129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.398381 17107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:01.398463 17107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "0059864cab15436b886044511cb5004a"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "e48b3088ac15025ae649a3776927eda9"
server_key_iv: "776936030791529be983b58c8f92b10d"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.398545 17107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:01.411772 17107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:01.413295 17107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:01.413431 17107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:01.413666 17107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:01.413933 17107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:01.413980 17107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.414014 17107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:01.414038 17107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.419190 17107 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:40169
I20260808 22:54:01.419296 17317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:40169 every 8 connection(s)
I20260808 22:54:01.419556 17107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:54:01.420464 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 17107
I20260808 22:54:01.420570 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:54:01.420778 22648 external_mini_cluster.cc:1428] Setting key cea11aa2863f2870cc63895d430dc783
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:01.428898 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.420167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:33917 (local address 127.22.30.62:38631)
0808 22:54:01.420206 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.420208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.420378 (+ 170us) server_negotiation.cc:407] Connection header received
0808 22:54:01.420884 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.420886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.420897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.420950 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:01.421910 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.422426 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.423685 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.423879 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.423938 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.426091 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:01.426092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:01.426093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:01.426101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:01.427525 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.427929 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.427931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.427931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.428001 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.428248 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.428248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.428249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.428378 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:01.428448 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.428718 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.428812 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:54:01.429104 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.420129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38631 (local address 127.22.30.2:33917)
0808 22:54:01.420260 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.420303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.420789 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.421007 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.421011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.421661 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:01.421823 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.421828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.422518 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.422519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.423595 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.423598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.423961 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.423962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.424101 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.424543 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:01.424552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:01.425999 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:01.427613 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:01.427620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:01.427622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:01.427846 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:01.428071 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:01.428071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:01.428072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:01.428170 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:01.428439 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:01.428446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:01.428585 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.428679 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.428846 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260808 22:54:01.429574 17318 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38631
I20260808 22:54:01.429667 17318 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
I20260808 22:54:01.429836 17318 heartbeater.cc:511] Master 127.22.30.62:38631 requested a full tablet report, sending...
I20260808 22:54:01.430190 16802 ts_manager.cc:194] Registered new tserver with Master: 0059864cab15436b886044511cb5004a (127.22.30.2:40169)
I20260808 22:54:01.430550 16802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:33917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:54:01.445257 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38631
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:01.522298 17327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:01.522457 17327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:01.522477 17327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:01.523475 17327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:01.523509 17327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:01.523533 17327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:54:01.524811 17327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:38631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:01.524959 17327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:01.525102 17327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:01.529661 17343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.529660 17335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.529714 17334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.529769 17327 server_base.cc:1034] running on GCE node
I20260808 22:54:01.530032 17327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:01.530239 17327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:01.531400 17327 hybrid_clock.cc:648] HybridClock initialized: now 1786229641531388 us; error 26 us; skew 500 ppm
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:01.533628 17327 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:54:01.534096 17327 webserver.cc:460] Webserver started at http://127.22.30.3:33343/ using document root <none> and password file <none>
I20260808 22:54:01.534299 17327 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:01.534358 17327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:01.534494 17327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:01.535362 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "62c99299de3c4af8a24ec4a67d7760e4"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "e7de9435764df8daf7c18872927ea38e"
server_key_iv: "f8b94b2f5a7967f5ea9640a13aece518"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.535789 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "62c99299de3c4af8a24ec4a67d7760e4"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "e7de9435764df8daf7c18872927ea38e"
server_key_iv: "f8b94b2f5a7967f5ea9640a13aece518"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.537002 17327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:01.537639 17352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.537808 17327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:01.537906 17327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "62c99299de3c4af8a24ec4a67d7760e4"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "e7de9435764df8daf7c18872927ea38e"
server_key_iv: "f8b94b2f5a7967f5ea9640a13aece518"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.537976 17327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:01.562216 17327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:01.564378 17327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:01.564527 17327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:01.564731 17327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:01.565058 17327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:01.565126 17327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.565160 17327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:01.565191 17327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.570851 17327 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:46047
I20260808 22:54:01.570902 17542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:46047 every 8 connection(s)
I20260808 22:54:01.571151 17327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:54:01 dist-test-slave-x4qh krb5kdc[16734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229641, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:01.578806 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.571762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:57201 (local address 127.22.30.62:38631)
0808 22:54:01.571829 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.571832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.571985 (+ 153us) server_negotiation.cc:407] Connection header received
0808 22:54:01.572453 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.572455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.572465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.572514 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:01.573323 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.573782 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.574621 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.574805 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.574879 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.576605 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:01.576607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:01.576607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:01.576615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:01.577604 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.577941 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.577942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.577943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.577997 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.578228 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.578229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.578229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.578336 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:01.578404 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.578645 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.578723 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:54:01.578950 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.571717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:38631 (local address 127.22.30.3:57201)
0808 22:54:01.571882 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.571919 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.572345 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.572568 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.572571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.573068 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:01.573224 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.573228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.573865 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.573867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.574534 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.574536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.574892 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.574893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.574996 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.575331 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:01.575337 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:01.576518 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:01.577676 (+ 1158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:01.577682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:01.577684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:01.577874 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:01.578054 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:01.578055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:01.578056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:01.578162 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:01.578398 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:01.578404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:01.578522 (+ 118us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.578609 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.578744 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260808 22:54:01.579447 17543 heartbeater.cc:347] Connected to a master server at 127.22.30.62:38631
I20260808 22:54:01.579514 17543 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:01.579702 17543 heartbeater.cc:511] Master 127.22.30.62:38631 requested a full tablet report, sending...
I20260808 22:54:01.579732 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 17327
I20260808 22:54:01.579847 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:54:01.580031 22648 external_mini_cluster.cc:1428] Setting key cdf4be1f5c67d2f0ddeba258b85489a4
I20260808 22:54:01.580045 16802 ts_manager.cc:194] Registered new tserver with Master: 62c99299de3c4af8a24ec4a67d7760e4 (127.22.30.3:46047)
I20260808 22:54:01.580361 16802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:57201
I20260808 22:54:01.580822 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:54:01.587081 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.582135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.22.30.62:38631)
0808 22:54:01.582177 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.582179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.582287 (+ 108us) server_negotiation.cc:407] Connection header received
0808 22:54:01.582321 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.582322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.582328 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.582367 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:01.582820 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.583176 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.583901 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.584053 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.584099 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.585123 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:01.585124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:01.585124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:01.585128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:01.586169 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.586449 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.586450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.586450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.586494 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:01.586602 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:01.586603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:01.586603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:01.586721 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:01.586783 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.586928 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.586995 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 22:54:01.588503 16802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:54:01.589874 16802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:54:01.595552 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.592018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.22.30.1:34151)
0808 22:54:01.592250 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.592253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.592296 (+ 43us) server_negotiation.cc:407] Connection header received
0808 22:54:01.592409 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.592412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.592427 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.592494 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:01.592713 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.593735 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.594638 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.595149 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.595221 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.595273 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.595435 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.595474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
I20260808 22:54:01.595552 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.591906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:34151 (local address 127.0.0.1:51510)
0808 22:54:01.592231 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.592241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.592305 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.592599 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.592600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.592602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:01.592638 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.592639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.593830 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.593831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.594558 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.594560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.595224 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.595224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.595306 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.595317 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.595417 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.595475 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260808 22:54:01.595716 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.591810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.22.30.3:46047)
0808 22:54:01.592205 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.592210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.592217 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:01.592250 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.592252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.592265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.592344 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:01.592500 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.593288 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.594593 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.595174 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.595234 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.595276 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.595609 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.595645 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":72,"threads_started":1}
I20260808 22:54:01.595829 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.592046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.22.30.2:40169)
0808 22:54:01.592209 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.592213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.592223 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:54:01.592266 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.592268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.592282 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.592522 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:01.592600 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.593576 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.594748 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.595350 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.595623 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.595684 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.595706 (+ 22us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.595751 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260808 22:54:01.595552 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.591749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:46047 (local address 127.0.0.1:56856)
0808 22:54:01.592024 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.592035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.592102 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.592369 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.592371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.592373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:01.592430 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.592432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.593393 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.593394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.594512 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.594515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.595309 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.595310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.595364 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.595391 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.595432 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.595472 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":43,"threads_started":1}
I20260808 22:54:01.595922 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.591717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40169 (local address 127.0.0.1:40300)
0808 22:54:01.591968 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.592000 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.592057 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.592376 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.592377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.592378 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:01.592451 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.592453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.593695 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.593696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.594670 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.594672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.595452 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.595453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.595506 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.595509 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.595534 (+ 25us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.595580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":35,"thread_start_us":45,"threads_started":1}
I20260808 22:54:01.596632 17215 tablet_service.cc:1467] Processing CreateTablet for tablet 4488d0251b2a450da05ce3ef6a22bd97 (DEFAULT_TABLE table=test-table [id=e7bfb7fc20c24fdaa9303a871c4fcc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:01.596632 17441 tablet_service.cc:1467] Processing CreateTablet for tablet 4488d0251b2a450da05ce3ef6a22bd97 (DEFAULT_TABLE table=test-table [id=e7bfb7fc20c24fdaa9303a871c4fcc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:01.596714 17003 tablet_service.cc:1467] Processing CreateTablet for tablet 4488d0251b2a450da05ce3ef6a22bd97 (DEFAULT_TABLE table=test-table [id=e7bfb7fc20c24fdaa9303a871c4fcc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:01.596954 17441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488d0251b2a450da05ce3ef6a22bd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:01.596954 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488d0251b2a450da05ce3ef6a22bd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:01.597065 17003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488d0251b2a450da05ce3ef6a22bd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:01.599193 17566 tablet_bootstrap.cc:492] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0: Bootstrap starting.
I20260808 22:54:01.599951 17568 tablet_bootstrap.cc:492] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Bootstrap starting.
I20260808 22:54:01.600107 17566 tablet_bootstrap.cc:654] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:01.600652 17568 tablet_bootstrap.cc:654] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:01.600665 17566 log.cc:824] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:01.601238 17568 log.cc:824] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:01.601459 17566 tablet_bootstrap.cc:492] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0: No bootstrap required, opened a new log
I20260808 22:54:01.601532 17566 ts_tablet_manager.cc:1397] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:01.601547 17569 tablet_bootstrap.cc:492] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Bootstrap starting.
I20260808 22:54:01.601908 17568 tablet_bootstrap.cc:492] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: No bootstrap required, opened a new log
I20260808 22:54:01.601972 17568 ts_tablet_manager.cc:1397] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:01.602159 17569 tablet_bootstrap.cc:654] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:01.602639 17569 log.cc:824] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:01.603104 17566 raft_consensus.cc:348] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.603286 17566 raft_consensus.cc:374] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:01.603332 17566 raft_consensus.cc:729] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8834ae230745e9afea58390cc359c0, State: Initialized, Role: FOLLOWER
I20260808 22:54:01.603456 17566 consensus_queue.cc:260] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.603734 17566 ts_tablet_manager.cc:1428] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:01.603757 17101 heartbeater.cc:503] Master 127.22.30.62:38631 was elected leader, sending a full tablet report...
I20260808 22:54:01.603992 17569 tablet_bootstrap.cc:492] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: No bootstrap required, opened a new log
I20260808 22:54:01.604102 17569 ts_tablet_manager.cc:1397] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:54:01.603986 17568 raft_consensus.cc:348] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.604140 17568 raft_consensus.cc:374] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:01.604188 17568 raft_consensus.cc:729] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0059864cab15436b886044511cb5004a, State: Initialized, Role: FOLLOWER
I20260808 22:54:01.604346 17568 consensus_queue.cc:260] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.604604 17568 ts_tablet_manager.cc:1428] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:01.604628 17318 heartbeater.cc:503] Master 127.22.30.62:38631 was elected leader, sending a full tablet report...
I20260808 22:54:01.605608 17569 raft_consensus.cc:348] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.605728 17569 raft_consensus.cc:374] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:01.605749 17569 raft_consensus.cc:729] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c99299de3c4af8a24ec4a67d7760e4, State: Initialized, Role: FOLLOWER
I20260808 22:54:01.605880 17569 consensus_queue.cc:260] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.606115 17569 ts_tablet_manager.cc:1428] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:01.606168 17543 heartbeater.cc:503] Master 127.22.30.62:38631 was elected leader, sending a full tablet report...
W20260808 22:54:01.670140 17321 tablet.cc:2369] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:54:01.694219 17573 raft_consensus.cc:482] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:01.694329 17573 raft_consensus.cc:504] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.694694 17573 leader_election.cc:288] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0059864cab15436b886044511cb5004a (127.22.30.2:40169), 62c99299de3c4af8a24ec4a67d7760e4 (127.22.30.3:46047)
I20260808 22:54:01.697427 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.694916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:38473 (local address 127.22.30.2:40169)
0808 22:54:01.694955 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.694956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.694964 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:01.694993 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.694994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.695005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.695068 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:01.695260 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.695799 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.696729 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.697081 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.697131 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.697139 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.697320 (+ 181us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.697362 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:54:01.697466 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.694910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:42719 (local address 127.22.30.3:46047)
0808 22:54:01.694949 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.694951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.695097 (+ 146us) server_negotiation.cc:407] Connection header received
0808 22:54:01.695131 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.695133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.695164 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.695225 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:01.695420 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.695893 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.696784 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.697135 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.697201 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.697210 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.697362 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.697406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:01.697456 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.694838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40169 (local address 127.22.30.1:38473)
0808 22:54:01.694883 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.694895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.694940 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.695086 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.695087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.695089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:01.695181 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.695183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.695874 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.695875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.696624 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.696626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.697157 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.697158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.697236 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.697239 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.697292 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.697352 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 22:54:01.697466 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.694854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:46047 (local address 127.22.30.1:42719)
0808 22:54:01.695025 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.695035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:01.695090 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:01.695259 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:01.695260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:01.695262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:01.695329 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.695330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.695971 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.695972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.696655 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:01.696657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.697217 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.697218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:01.697267 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.697270 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:01.697336 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:54:01.697370 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260808 22:54:01.697862 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488d0251b2a450da05ce3ef6a22bd97" candidate_uuid: "fa8834ae230745e9afea58390cc359c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0059864cab15436b886044511cb5004a" is_pre_election: true
I20260808 22:54:01.697984 17472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488d0251b2a450da05ce3ef6a22bd97" candidate_uuid: "fa8834ae230745e9afea58390cc359c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c99299de3c4af8a24ec4a67d7760e4" is_pre_election: true
I20260808 22:54:01.698024 17242 raft_consensus.cc:2454] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8834ae230745e9afea58390cc359c0 in term 0.
I20260808 22:54:01.698136 17472 raft_consensus.cc:2454] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8834ae230745e9afea58390cc359c0 in term 0.
I20260808 22:54:01.698252 16932 leader_election.cc:302] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0059864cab15436b886044511cb5004a, fa8834ae230745e9afea58390cc359c0; no voters:
I20260808 22:54:01.698385 17573 raft_consensus.cc:2781] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:01.698450 17573 raft_consensus.cc:482] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:01.698477 17573 raft_consensus.cc:3037] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:01.699160 17573 raft_consensus.cc:504] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.699285 17573 leader_election.cc:288] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [CANDIDATE]: Term 1 election: Requested vote from peers 0059864cab15436b886044511cb5004a (127.22.30.2:40169), 62c99299de3c4af8a24ec4a67d7760e4 (127.22.30.3:46047)
I20260808 22:54:01.699450 17472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488d0251b2a450da05ce3ef6a22bd97" candidate_uuid: "fa8834ae230745e9afea58390cc359c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c99299de3c4af8a24ec4a67d7760e4"
I20260808 22:54:01.699523 17472 raft_consensus.cc:3037] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:01.699519 17242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488d0251b2a450da05ce3ef6a22bd97" candidate_uuid: "fa8834ae230745e9afea58390cc359c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0059864cab15436b886044511cb5004a"
I20260808 22:54:01.699606 17242 raft_consensus.cc:3037] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:01.700237 17472 raft_consensus.cc:2454] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8834ae230745e9afea58390cc359c0 in term 1.
I20260808 22:54:01.700297 17242 raft_consensus.cc:2454] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8834ae230745e9afea58390cc359c0 in term 1.
I20260808 22:54:01.700543 16931 leader_election.cc:302] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c99299de3c4af8a24ec4a67d7760e4, fa8834ae230745e9afea58390cc359c0; no voters:
I20260808 22:54:01.700635 17573 raft_consensus.cc:2781] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:01.700822 17573 raft_consensus.cc:686] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [term 1 LEADER]: Becoming Leader. State: Replica: fa8834ae230745e9afea58390cc359c0, State: Running, Role: LEADER
I20260808 22:54:01.700922 17573 consensus_queue.cc:237] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } }
I20260808 22:54:01.701725 16802 catalog_manager.cc:5495] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa8834ae230745e9afea58390cc359c0 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "fa8834ae230745e9afea58390cc359c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8834ae230745e9afea58390cc359c0" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 34151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 } health_report { overall_health: UNKNOWN } } }
I20260808 22:54:01.708835 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.706712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.22.30.1:34151)
0808 22:54:01.706768 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.706770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.706780 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:54:01.706861 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.706861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.706878 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.706934 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:01.707063 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.707474 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.708153 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.708342 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.708404 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.708565 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:01.708655 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:01.708735 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.708756 (+ 21us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.708774 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:54:01.710183 17472 raft_consensus.cc:1261] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 1 FOLLOWER]: Refusing update from remote peer fa8834ae230745e9afea58390cc359c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:01.710198 17242 raft_consensus.cc:1261] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 1 FOLLOWER]: Refusing update from remote peer fa8834ae230745e9afea58390cc359c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:01.710489 17578 consensus_queue.cc:1035] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c99299de3c4af8a24ec4a67d7760e4" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 46047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:01.710588 17573 consensus_queue.cc:1035] T 4488d0251b2a450da05ce3ef6a22bd97 P fa8834ae230745e9afea58390cc359c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0059864cab15436b886044511cb5004a" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:01.712738 17587 mvcc.cc:205] Tried to move back new op lower bound from 7316396612442886144 to 7316396612407623680. Current Snapshot: MvccSnapshot[applied={T|T < 7316396612442886144}]
I20260808 22:54:01.715162 16802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:01.715256 16802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:01.715975 16802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea37340cdb484e63ace39d43e023976b: Sending DeleteTablet for 3 replicas of tablet 4488d0251b2a450da05ce3ef6a22bd97
I20260808 22:54:01.716271 17215 tablet_service.cc:1514] Processing DeleteTablet for tablet 4488d0251b2a450da05ce3ef6a22bd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:01 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:40300
I20260808 22:54:01.716305 17441 tablet_service.cc:1514] Processing DeleteTablet for tablet 4488d0251b2a450da05ce3ef6a22bd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:01 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56856
I20260808 22:54:01.716514 17597 tablet_replica.cc:330] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: stopping tablet replica
I20260808 22:54:01.716516 17003 tablet_service.cc:1514] Processing DeleteTablet for tablet 4488d0251b2a450da05ce3ef6a22bd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:01 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:51510
I20260808 22:54:01.716583 17599 tablet_replica.cc:330] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: stopping tablet replica
I20260808 22:54:01.716671 17597 raft_consensus.cc:2229] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:01.716739 17599 raft_consensus.cc:2229] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:01.716823 17597 raft_consensus.cc:2258] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:01.716941 17599 raft_consensus.cc:2258] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:01.716971 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 16910
I20260808 22:54:01.717630 17597 ts_tablet_manager.cc:1905] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:01.717661 17599 ts_tablet_manager.cc:1905] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:01.719312 17597 ts_tablet_manager.cc:1918] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:01.719398 17597 log.cc:1196] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/4488d0251b2a450da05ce3ef6a22bd97
I20260808 22:54:01.719364 17599 ts_tablet_manager.cc:1918] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:01.719473 17599 log.cc:1196] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/4488d0251b2a450da05ce3ef6a22bd97
I20260808 22:54:01.719640 17597 ts_tablet_manager.cc:1939] T 4488d0251b2a450da05ce3ef6a22bd97 P 0059864cab15436b886044511cb5004a: Deleting consensus metadata
I20260808 22:54:01.719780 17599 ts_tablet_manager.cc:1939] T 4488d0251b2a450da05ce3ef6a22bd97 P 62c99299de3c4af8a24ec4a67d7760e4: Deleting consensus metadata
I20260808 22:54:01.719995 16778 catalog_manager.cc:4841] TS 0059864cab15436b886044511cb5004a (127.22.30.2:40169): tablet 4488d0251b2a450da05ce3ef6a22bd97 (table test-table [id=e7bfb7fc20c24fdaa9303a871c4fcc4a]) successfully deleted
I20260808 22:54:01.720129 16777 catalog_manager.cc:4841] TS 62c99299de3c4af8a24ec4a67d7760e4 (127.22.30.3:46047): tablet 4488d0251b2a450da05ce3ef6a22bd97 (table test-table [id=e7bfb7fc20c24fdaa9303a871c4fcc4a]) successfully deleted
W20260808 22:54:01.727880 16778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:34151 (error 108)
I20260808 22:54:01.728134 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 17107
I20260808 22:54:01.728329 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.728218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:34151 (local address 127.0.0.1:51520)
0808 22:54:01.728261 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:01.728295 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 22:54:01.730177 16778 catalog_manager.cc:4584] TS fa8834ae230745e9afea58390cc359c0 (127.22.30.1:34151): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4488d0251b2a450da05ce3ef6a22bd97: Network error: Client connection negotiation failed: client connection to 127.22.30.1:34151: connect: Connection refused (error 111)
I20260808 22:54:01.738305 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 17327
I20260808 22:54:01.745749 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 16753
2026-08-08T22:54:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2048 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T22:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:01Z Disabled control of system clock
I20260808 22:54:01.776060 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:36283
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:01.852552 17605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:01.852730 17605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:01.852747 17605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:01.853765 17605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:01.853802 17605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:01.853813 17605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:01.853825 17605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:01.853868 17605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:01.855047 17605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:36283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:01.855245 17605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:01.855572 17605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:01.859826 17617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:01.859848 17616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.859877 17605 server_base.cc:1034] running on GCE node
W20260808 22:54:01.859848 17620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:01.860292 17605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:01.860539 17605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:01.861670 17605 hybrid_clock.cc:648] HybridClock initialized: now 1786229641861656 us; error 28 us; skew 500 ppm
I20260808 22:54:01.862916 17605 webserver.cc:460] Webserver started at http://127.22.30.62:42607/ using document root <none> and password file <none>
I20260808 22:54:01.863137 17605 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:01.863181 17605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:01.863324 17605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:01.864279 17605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "d19ea6c223624f5daee3dc54fde04986"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "238d7eafbd6dce6a0ea8ecce7c2af4d3"
server_key_iv: "15d0bc8dffd71788a75bb0ba24e4208f"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.864630 17605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "d19ea6c223624f5daee3dc54fde04986"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "238d7eafbd6dce6a0ea8ecce7c2af4d3"
server_key_iv: "15d0bc8dffd71788a75bb0ba24e4208f"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.865944 17605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:01.866605 17633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:01.866765 17605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:01.866860 17605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d19ea6c223624f5daee3dc54fde04986"
format_stamp: "Formatted at 2026-08-08 22:54:01 on dist-test-slave-x4qh"
server_key: "238d7eafbd6dce6a0ea8ecce7c2af4d3"
server_key_iv: "15d0bc8dffd71788a75bb0ba24e4208f"
server_key_version: "encryptionkey@0"
I20260808 22:54:01.866936 17605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:01.888464 17605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:01.888746 17605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:01.888903 17605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:01.892560 17605 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:36283
I20260808 22:54:01.892585 17721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:36283 every 8 connection(s)
I20260808 22:54:01.892884 17605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:01.893653 17723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:01.895587 17723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: Bootstrap starting.
I20260808 22:54:01.896132 17723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:01.896373 17723 log.cc:824] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:01.896993 17723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: No bootstrap required, opened a new log
I20260808 22:54:01.898461 17723 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ea6c223624f5daee3dc54fde04986" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36283 } }
I20260808 22:54:01.898579 17723 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:01.898599 17723 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19ea6c223624f5daee3dc54fde04986, State: Initialized, Role: FOLLOWER
I20260808 22:54:01.898715 17723 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ea6c223624f5daee3dc54fde04986" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36283 } }
I20260808 22:54:01.898787 17723 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:01.898808 17723 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:01.898833 17723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:01.899241 17723 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ea6c223624f5daee3dc54fde04986" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36283 } }
I20260808 22:54:01.899319 17723 leader_election.cc:302] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19ea6c223624f5daee3dc54fde04986; no voters:
I20260808 22:54:01.899475 17723 leader_election.cc:288] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:01.899531 17728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:01.899698 17723 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:01.899722 17728 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [term 1 LEADER]: Becoming Leader. State: Replica: d19ea6c223624f5daee3dc54fde04986, State: Running, Role: LEADER
I20260808 22:54:01.899801 17728 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ea6c223624f5daee3dc54fde04986" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36283 } }
I20260808 22:54:01.900238 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 17605
I20260808 22:54:01.900364 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:01.900604 22648 external_mini_cluster.cc:1428] Setting key 09a754859747e4402482c6e45600def9
I20260808 22:54:01.901010 17731 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19ea6c223624f5daee3dc54fde04986. Latest consensus state: current_term: 1 leader_uuid: "d19ea6c223624f5daee3dc54fde04986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ea6c223624f5daee3dc54fde04986" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36283 } } }
I20260808 22:54:01.901146 17730 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19ea6c223624f5daee3dc54fde04986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ea6c223624f5daee3dc54fde04986" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36283 } } }
I20260808 22:54:01.901571 17730 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:01.901365 17731 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986 [sys.catalog]: This master's current role is: LEADER
W20260808 22:54:01.901665 17762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:01.901703 17762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:01.901859 17767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:01.902053 17767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:01.903395 17767 catalog_manager.cc:1347] Generated new cluster ID: 808fc77ba25e4cecb776f96c9da3131b
I20260808 22:54:01.903443 17767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:01.905800 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:01.901233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.22.30.62:36283)
0808 22:54:01.901451 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:01.901460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:01.901498 (+ 38us) server_negotiation.cc:407] Connection header received
0808 22:54:01.901604 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:01.901611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:01.901690 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:01.901843 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:01.902053 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.903114 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.903943 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:01.904376 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:01.904510 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:01.905143 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:01.905146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:01.905147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:01.905319 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:01.905349 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:01.905377 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:01.905454 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:01.905532 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 22:54:01.905611 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260808 22:54:01.909075 17767 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:01.909653 17767 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:01.912835 17767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: Generated new TSK 0
I20260808 22:54:01.912987 17767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:01.920848 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36283
--builtin_ntp_servers=127.22.30.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:02.019065 17771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:02.019248 17771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:02.019268 17771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:02.020295 17771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:02.020330 17771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:02.020354 17771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:54:02.021531 17771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:02.021728 17771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:02.021989 17771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:02.027750 17784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:02.027765 17782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:02.027783 17786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:02.027949 17771 server_base.cc:1034] running on GCE node
I20260808 22:54:02.028211 17771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:02.028488 17771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:02.029670 17771 hybrid_clock.cc:648] HybridClock initialized: now 1786229642029649 us; error 36 us; skew 500 ppm
I20260808 22:54:02.031057 17771 webserver.cc:460] Webserver started at http://127.22.30.1:42765/ using document root <none> and password file <none>
I20260808 22:54:02.031334 17771 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:02.031430 17771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:02.031569 17771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:02.032895 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "5b12d2e750f24aae88a6691e89dbcd1b"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "35a21f663b81584277bd55407c0ff5c4"
server_key_iv: "93bfff38c4002f39514eaa0bca2db406"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.033294 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "5b12d2e750f24aae88a6691e89dbcd1b"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "35a21f663b81584277bd55407c0ff5c4"
server_key_iv: "93bfff38c4002f39514eaa0bca2db406"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.034897 17771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:02.035760 17796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.035948 17771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:54:02.036043 17771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "5b12d2e750f24aae88a6691e89dbcd1b"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "35a21f663b81584277bd55407c0ff5c4"
server_key_iv: "93bfff38c4002f39514eaa0bca2db406"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.036128 17771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:02.055502 17771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:02.055816 17771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:02.055951 17771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:02.056156 17771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:02.056469 17771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:02.056512 17771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.056538 17771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:02.056561 17771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.062939 17771 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:35917
I20260808 22:54:02.063002 17975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:35917 every 8 connection(s)
I20260808 22:54:02.063284 17771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:54:02.066884 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 17771
I20260808 22:54:02.067001 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:54:02.067191 22648 external_mini_cluster.cc:1428] Setting key 1f88354c11ab72685d977f6a5625dfee
I20260808 22:54:02.068177 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36283
--builtin_ntp_servers=127.22.30.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:54:02.068542 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.064081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:55199 (local address 127.22.30.62:36283)
0808 22:54:02.064137 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.064139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.064377 (+ 238us) server_negotiation.cc:407] Connection header received
0808 22:54:02.064888 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.064890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.064903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.064959 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:02.065613 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.066109 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.067081 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.067284 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.067326 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.068158 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:02.068159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:02.068159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:02.068171 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:02.068185 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:02.068189 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:02.068233 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.068437 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.068469 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:54:02.068733 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.064083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36283 (local address 127.22.30.1:55199)
0808 22:54:02.064256 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.064285 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.064775 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.065036 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.065039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.065047 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:02.065501 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.065505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.066206 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.066208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.066953 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.066956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.067459 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.067459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.067581 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.068014 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:02.068015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:02.068039 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:02.068059 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:02.068354 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:02.068360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:02.068362 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.068437 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.068528 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260808 22:54:02.069169 17976 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36283
I20260808 22:54:02.069262 17976 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:02.069432 17976 heartbeater.cc:511] Master 127.22.30.62:36283 requested a full tablet report, sending...
I20260808 22:54:02.069928 17660 ts_manager.cc:194] Registered new tserver with Master: 5b12d2e750f24aae88a6691e89dbcd1b (127.22.30.1:35917)
I20260808 22:54:02.070468 17660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.1:55199
W20260808 22:54:02.144163 17980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:02.144346 17980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:02.144367 17980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:02.145395 17980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:02.145426 17980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:02.145449 17980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:54:02.146651 17980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:02.146828 17980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:02.147032 17980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:02.151358 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:02.151412 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:02.151384 17994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:02.151657 17980 server_base.cc:1034] running on GCE node
I20260808 22:54:02.151903 17980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:02.152120 17980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:02.153264 17980 hybrid_clock.cc:648] HybridClock initialized: now 1786229642153236 us; error 30 us; skew 500 ppm
I20260808 22:54:02.154485 17980 webserver.cc:460] Webserver started at http://127.22.30.2:45821/ using document root <none> and password file <none>
I20260808 22:54:02.154682 17980 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:02.154726 17980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:02.154811 17980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:02.155792 17980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "6e2bc30ac2c1478da58fe597a7a2b70d"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "8b226e3195ec82be658ac6e0c5e0de6a"
server_key_iv: "a3d7f6e2b14b7802b7856ea71cc65d14"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.156076 17980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "6e2bc30ac2c1478da58fe597a7a2b70d"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "8b226e3195ec82be658ac6e0c5e0de6a"
server_key_iv: "a3d7f6e2b14b7802b7856ea71cc65d14"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.157231 17980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:02.157822 18000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.157976 17980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:02.158061 17980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "6e2bc30ac2c1478da58fe597a7a2b70d"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "8b226e3195ec82be658ac6e0c5e0de6a"
server_key_iv: "a3d7f6e2b14b7802b7856ea71cc65d14"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.158128 17980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:02.171185 17980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:02.171455 17980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:02.171595 17980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:02.171782 17980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:02.172051 17980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:02.172099 17980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.172142 17980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:02.172164 17980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.177625 17980 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:35361
I20260808 22:54:02.177655 18194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:35361 every 8 connection(s)
I20260808 22:54:02.177927 17980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:54:02.182274 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.178518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:41729 (local address 127.22.30.62:36283)
0808 22:54:02.178575 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.178577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.178755 (+ 178us) server_negotiation.cc:407] Connection header received
0808 22:54:02.179165 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.179166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.179175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.179226 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:02.179849 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.180305 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.181155 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.181344 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.181391 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.182089 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:02.182090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:02.182091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:02.182094 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:02.182105 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:02.182109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:02.182157 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.182201 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.182221 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:54:02.182466 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.178499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36283 (local address 127.22.30.2:41729)
0808 22:54:02.178652 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.178684 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.179079 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.179269 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.179271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.179285 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:02.179777 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.179783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.180397 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.180398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.181086 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.181087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.181414 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.181415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.181511 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.181977 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:02.181977 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:02.182001 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:02.182022 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:02.182157 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:02.182159 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:02.182161 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.182204 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.182271 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260808 22:54:02.182911 18195 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36283
I20260808 22:54:02.182991 18195 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:02.183117 18195 heartbeater.cc:511] Master 127.22.30.62:36283 requested a full tablet report, sending...
I20260808 22:54:02.183460 17660 ts_manager.cc:194] Registered new tserver with Master: 6e2bc30ac2c1478da58fe597a7a2b70d (127.22.30.2:35361)
I20260808 22:54:02.183686 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 17980
I20260808 22:54:02.183794 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:54:02.183864 17660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.2:41729
I20260808 22:54:02.184019 22648 external_mini_cluster.cc:1428] Setting key a108441bbfc6a8944fa0eccaefcaf440
I20260808 22:54:02.184902 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36283
--builtin_ntp_servers=127.22.30.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:02.259570 18199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:02.259761 18199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:02.259784 18199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:02.260905 18199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:02.260942 18199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:02.260993 18199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:54:02.262295 18199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:02.262497 18199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:02.262737 18199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:02.267402 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:02.267409 18199 server_base.cc:1034] running on GCE node
W20260808 22:54:02.267416 18207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:02.267438 18210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:02.267815 18199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:02.268033 18199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:02.269166 18199 hybrid_clock.cc:648] HybridClock initialized: now 1786229642269152 us; error 30 us; skew 500 ppm
I20260808 22:54:02.270360 18199 webserver.cc:460] Webserver started at http://127.22.30.3:45803/ using document root <none> and password file <none>
I20260808 22:54:02.270576 18199 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:02.270617 18199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:02.270743 18199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:02.271711 18199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "340ea44366be4c1490249b1d8758d9a7"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "033ea683c56b1a4aa9feb8dfbd4c53e8"
server_key_iv: "fdc3cd37b24d40ca697ac5f923e3d2bc"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.272032 18199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "340ea44366be4c1490249b1d8758d9a7"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "033ea683c56b1a4aa9feb8dfbd4c53e8"
server_key_iv: "fdc3cd37b24d40ca697ac5f923e3d2bc"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.273245 18199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:02.273904 18218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.274065 18199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:02.274154 18199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "340ea44366be4c1490249b1d8758d9a7"
format_stamp: "Formatted at 2026-08-08 22:54:02 on dist-test-slave-x4qh"
server_key: "033ea683c56b1a4aa9feb8dfbd4c53e8"
server_key_iv: "fdc3cd37b24d40ca697ac5f923e3d2bc"
server_key_version: "encryptionkey@0"
I20260808 22:54:02.274235 18199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:02.287117 18199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:02.287336 18199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:02.287492 18199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:02.287698 18199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:02.287967 18199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:02.288014 18199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.288050 18199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:02.288075 18199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:02.293538 18199 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:45315
I20260808 22:54:02.293596 18399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:45315 every 8 connection(s)
I20260808 22:54:02.293855 18199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:54:02.298086 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.294421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:38529 (local address 127.22.30.62:36283)
0808 22:54:02.294472 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.294474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.294645 (+ 171us) server_negotiation.cc:407] Connection header received
0808 22:54:02.295039 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.295040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.295050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.295097 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:02.295723 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.296184 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.297063 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.297235 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.297285 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.297892 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:02.297893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:02.297893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:02.297896 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:02.297908 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:02.297911 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:02.297956 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.298012 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.298033 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 22:54:02.298231 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.294414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36283 (local address 127.22.30.3:38529)
0808 22:54:02.294557 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.294583 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.294955 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.295138 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.295141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.295148 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:02.295651 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.295656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.296256 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.296257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.296998 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.297000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.297315 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.297316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.297410 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.297780 (+ 370us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:02.297781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:02.297804 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:02.297824 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:02.297959 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:02.297961 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:02.297963 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.298004 (+ 41us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.298067 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260808 22:54:02.298689 18401 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36283
I20260808 22:54:02.298782 18401 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:02.298929 18401 heartbeater.cc:511] Master 127.22.30.62:36283 requested a full tablet report, sending...
I20260808 22:54:02.299260 17660 ts_manager.cc:194] Registered new tserver with Master: 340ea44366be4c1490249b1d8758d9a7 (127.22.30.3:45315)
I20260808 22:54:02.299638 17660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.3:38529
I20260808 22:54:02.300101 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 18199
I20260808 22:54:02.300213 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:54:02.300419 22648 external_mini_cluster.cc:1428] Setting key 29148ca9ef41306083d492f5976679c2
I20260808 22:54:02.301261 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:54:02.304812 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.302573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.22.30.62:36283)
0808 22:54:02.302625 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.302627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.302722 (+ 95us) server_negotiation.cc:407] Connection header received
0808 22:54:02.302763 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.302763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.302772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.302853 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:02.302966 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.303346 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.304010 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.304163 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.304206 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.304653 (+ 447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:02.304653 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:02.304654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:02.304657 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:02.304667 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:02.304670 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:02.304710 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.304745 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.304763 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:54:02.306210 17660 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:54:02.307617 17660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:54:02.313098 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.309396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35917 (local address 127.0.0.1:34078)
0808 22:54:02.309987 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.309995 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.310063 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.310363 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.310364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.310365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:02.310400 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.310401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.311183 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.311184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.312131 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.312134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.312819 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.312820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.312897 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.312909 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.312979 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.313025 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":37,"thread_start_us":81,"threads_started":1}
I20260808 22:54:02.313109 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.309644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.22.30.1:35917)
0808 22:54:02.309788 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.309792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.310060 (+ 268us) server_negotiation.cc:407] Connection header received
0808 22:54:02.310209 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.310212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.310227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.310292 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:02.310460 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.311100 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.312229 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.312729 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.312819 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.312902 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.312992 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.313034 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260808 22:54:02.313603 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.309396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:35361 (local address 127.0.0.1:41362)
0808 22:54:02.309508 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.309541 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.309594 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.310142 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.310145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.310147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:02.310204 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.310206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.311456 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.311457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.312502 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.312504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.313386 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.313387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.313453 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.313458 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.313508 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.313544 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260808 22:54:02.314106 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.309401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:45315 (local address 127.0.0.1:42870)
0808 22:54:02.310496 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.310505 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.310570 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.310755 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.310755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.310757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:02.310798 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.310800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.311957 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.311958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.312978 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.312980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.313896 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.313897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.313959 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.313962 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.314012 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.314048 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":85,"threads_started":1}
I20260808 22:54:02.314122 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.309589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.22.30.3:45315)
0808 22:54:02.309720 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.309723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.310547 (+ 824us) server_negotiation.cc:407] Connection header received
0808 22:54:02.310640 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.310642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.310654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.310696 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:02.310886 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.311871 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.313072 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.313718 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.313758 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.313835 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.314045 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.314073 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260808 22:54:02.313999 17868 tablet_service.cc:1467] Processing CreateTablet for tablet 91bab11c937748c49fd7cc7c9269823e (DEFAULT_TABLE table=test-table [id=d41a3957ac044ec2a0145c1865fe2e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:02.314352 17868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91bab11c937748c49fd7cc7c9269823e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:02.314996 18297 tablet_service.cc:1467] Processing CreateTablet for tablet 91bab11c937748c49fd7cc7c9269823e (DEFAULT_TABLE table=test-table [id=d41a3957ac044ec2a0145c1865fe2e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:02.315275 18297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91bab11c937748c49fd7cc7c9269823e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:02.315510 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.309832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.22.30.2:35361)
0808 22:54:02.309991 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.309993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.310001 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:02.310034 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.310035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.310045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.310124 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:02.310267 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.311346 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.312590 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.313295 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.313373 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.313441 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.315419 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.315457 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260808 22:54:02.317670 18430 tablet_bootstrap.cc:492] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b: Bootstrap starting.
I20260808 22:54:02.318223 18430 tablet_bootstrap.cc:654] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:02.318286 18431 tablet_bootstrap.cc:492] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Bootstrap starting.
I20260808 22:54:02.318576 18430 log.cc:824] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:02.318941 18431 tablet_bootstrap.cc:654] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:02.318897 18088 tablet_service.cc:1467] Processing CreateTablet for tablet 91bab11c937748c49fd7cc7c9269823e (DEFAULT_TABLE table=test-table [id=d41a3957ac044ec2a0145c1865fe2e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:02.319175 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91bab11c937748c49fd7cc7c9269823e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:02.319296 18431 log.cc:824] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:02.319391 18430 tablet_bootstrap.cc:492] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b: No bootstrap required, opened a new log
I20260808 22:54:02.319465 18430 ts_tablet_manager.cc:1397] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:02.319926 18431 tablet_bootstrap.cc:492] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: No bootstrap required, opened a new log
I20260808 22:54:02.320015 18431 ts_tablet_manager.cc:1397] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 22:54:02.321861 18431 raft_consensus.cc:348] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.321965 18431 raft_consensus.cc:374] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:02.321985 18431 raft_consensus.cc:729] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340ea44366be4c1490249b1d8758d9a7, State: Initialized, Role: FOLLOWER
I20260808 22:54:02.321926 18430 raft_consensus.cc:348] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.322075 18430 raft_consensus.cc:374] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:02.322110 18430 raft_consensus.cc:729] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b12d2e750f24aae88a6691e89dbcd1b, State: Initialized, Role: FOLLOWER
I20260808 22:54:02.322089 18431 consensus_queue.cc:260] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.322216 18430 consensus_queue.cc:260] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.322312 18431 ts_tablet_manager.cc:1428] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 22:54:02.322450 18401 heartbeater.cc:503] Master 127.22.30.62:36283 was elected leader, sending a full tablet report...
I20260808 22:54:02.322465 18435 tablet_bootstrap.cc:492] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Bootstrap starting.
I20260808 22:54:02.322780 18430 ts_tablet_manager.cc:1428] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:02.322896 17976 heartbeater.cc:503] Master 127.22.30.62:36283 was elected leader, sending a full tablet report...
I20260808 22:54:02.323033 18435 tablet_bootstrap.cc:654] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:02.323354 18435 log.cc:824] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:02.324095 18435 tablet_bootstrap.cc:492] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: No bootstrap required, opened a new log
I20260808 22:54:02.324169 18435 ts_tablet_manager.cc:1397] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 22:54:02.325992 18435 raft_consensus.cc:348] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.326121 18435 raft_consensus.cc:374] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:02.326165 18435 raft_consensus.cc:729] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2bc30ac2c1478da58fe597a7a2b70d, State: Initialized, Role: FOLLOWER
I20260808 22:54:02.326267 18435 consensus_queue.cc:260] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.326489 18435 ts_tablet_manager.cc:1428] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 22:54:02.326648 18195 heartbeater.cc:503] Master 127.22.30.62:36283 was elected leader, sending a full tablet report...
W20260808 22:54:02.428438 18197 tablet.cc:2369] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:54:02.512087 18441 raft_consensus.cc:482] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:02.512197 18441 raft_consensus.cc:504] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.512539 18441 leader_election.cc:288] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b12d2e750f24aae88a6691e89dbcd1b (127.22.30.1:35917), 340ea44366be4c1490249b1d8758d9a7 (127.22.30.3:45315)
I20260808 22:54:02.515048 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.512723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35917 (local address 127.22.30.2:54269)
0808 22:54:02.512800 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.512811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.512850 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.513009 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.513010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.513011 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:02.513091 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.513092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.513672 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.513673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.514339 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.514341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.514825 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.514826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.514890 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.514892 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.514944 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.514982 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 22:54:02.515048 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.512804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:54269 (local address 127.22.30.1:35917)
0808 22:54:02.512889 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.512891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.512900 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:02.512931 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.512932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.512941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.513001 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:02.513160 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.513593 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.514423 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.514737 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.514807 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.514818 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.514951 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.514990 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 22:54:02.515180 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.512824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57673 (local address 127.22.30.3:45315)
0808 22:54:02.512881 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.512883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.513019 (+ 136us) server_negotiation.cc:407] Connection header received
0808 22:54:02.513083 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.513084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.513093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.513151 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:02.513312 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.513728 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.514544 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.514862 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.514933 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.514942 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.515087 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.515124 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 22:54:02.515192 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.512759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:45315 (local address 127.22.30.2:57673)
0808 22:54:02.512948 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.512956 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:02.513025 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:02.513172 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:02.513173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:02.513175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:02.513241 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.513242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.513832 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.513834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.514460 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:02.514462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.514937 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.514938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:02.515021 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.515023 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:02.515080 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 22:54:02.515115 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260808 22:54:02.515508 17898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bab11c937748c49fd7cc7c9269823e" candidate_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" is_pre_election: true
I20260808 22:54:02.515579 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bab11c937748c49fd7cc7c9269823e" candidate_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340ea44366be4c1490249b1d8758d9a7" is_pre_election: true
I20260808 22:54:02.515697 17898 raft_consensus.cc:2454] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2bc30ac2c1478da58fe597a7a2b70d in term 0.
I20260808 22:54:02.515717 18325 raft_consensus.cc:2454] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2bc30ac2c1478da58fe597a7a2b70d in term 0.
I20260808 22:54:02.515929 18004 leader_election.cc:302] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340ea44366be4c1490249b1d8758d9a7, 6e2bc30ac2c1478da58fe597a7a2b70d; no voters:
I20260808 22:54:02.516067 18441 raft_consensus.cc:2781] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:02.516129 18441 raft_consensus.cc:482] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:02.516146 18441 raft_consensus.cc:3037] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:02.516866 18441 raft_consensus.cc:504] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.516985 18441 leader_election.cc:288] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [CANDIDATE]: Term 1 election: Requested vote from peers 5b12d2e750f24aae88a6691e89dbcd1b (127.22.30.1:35917), 340ea44366be4c1490249b1d8758d9a7 (127.22.30.3:45315)
I20260808 22:54:02.517135 17898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bab11c937748c49fd7cc7c9269823e" candidate_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b12d2e750f24aae88a6691e89dbcd1b"
I20260808 22:54:02.517140 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bab11c937748c49fd7cc7c9269823e" candidate_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340ea44366be4c1490249b1d8758d9a7"
I20260808 22:54:02.517196 18325 raft_consensus.cc:3037] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:02.517213 17898 raft_consensus.cc:3037] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:02.517884 18325 raft_consensus.cc:2454] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2bc30ac2c1478da58fe597a7a2b70d in term 1.
I20260808 22:54:02.517994 17898 raft_consensus.cc:2454] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2bc30ac2c1478da58fe597a7a2b70d in term 1.
I20260808 22:54:02.518028 18004 leader_election.cc:302] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340ea44366be4c1490249b1d8758d9a7, 6e2bc30ac2c1478da58fe597a7a2b70d; no voters:
I20260808 22:54:02.518118 18441 raft_consensus.cc:2781] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:02.518261 18441 raft_consensus.cc:686] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 1 LEADER]: Becoming Leader. State: Replica: 6e2bc30ac2c1478da58fe597a7a2b70d, State: Running, Role: LEADER
I20260808 22:54:02.518345 18441 consensus_queue.cc:237] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } }
I20260808 22:54:02.519116 17660 catalog_manager.cc:5495] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e2bc30ac2c1478da58fe597a7a2b70d (127.22.30.2). New cstate: current_term: 1 leader_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e2bc30ac2c1478da58fe597a7a2b70d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20260808 22:54:02.537519 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.535589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.22.30.2:35361)
0808 22:54:02.535662 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:02.535664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:02.535674 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:54:02.535738 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:02.535739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:02.535755 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:02.535824 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:02.535929 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.536334 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.536961 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:02.537124 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:02.537166 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:02.537267 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:02.537344 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:02.537395 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:02.537429 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 22:54:02.537441 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 22:54:02.539028 18325 raft_consensus.cc:1261] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 1 FOLLOWER]: Refusing update from remote peer 6e2bc30ac2c1478da58fe597a7a2b70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:02.539054 17898 raft_consensus.cc:1261] T 91bab11c937748c49fd7cc7c9269823e P 5b12d2e750f24aae88a6691e89dbcd1b [term 1 FOLLOWER]: Refusing update from remote peer 6e2bc30ac2c1478da58fe597a7a2b70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:02.539343 18443 consensus_queue.cc:1035] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b12d2e750f24aae88a6691e89dbcd1b" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 35917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:02.539458 18441 consensus_queue.cc:1035] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "340ea44366be4c1490249b1d8758d9a7" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:02.541666 18450 mvcc.cc:205] Tried to move back new op lower bound from 7316396615837569024 to 7316396615755751424. Current Snapshot: MvccSnapshot[applied={T|T < 7316396615837569024}]
I20260808 22:54:02.541864 18451 mvcc.cc:205] Tried to move back new op lower bound from 7316396615837569024 to 7316396615755751424. Current Snapshot: MvccSnapshot[applied={T|T < 7316396615837569024}]
I20260808 22:54:02.543828 17660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:02.543921 17660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:02.544652 17660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d19ea6c223624f5daee3dc54fde04986: Sending DeleteTablet for 3 replicas of tablet 91bab11c937748c49fd7cc7c9269823e
I20260808 22:54:02.544960 18297 tablet_service.cc:1514] Processing DeleteTablet for tablet 91bab11c937748c49fd7cc7c9269823e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:02 UTC) from {username='slave'} at 127.0.0.1:42870
I20260808 22:54:02.544967 17868 tablet_service.cc:1514] Processing DeleteTablet for tablet 91bab11c937748c49fd7cc7c9269823e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:02 UTC) from {username='slave'} at 127.0.0.1:34078
I20260808 22:54:02.544962 18088 tablet_service.cc:1514] Processing DeleteTablet for tablet 91bab11c937748c49fd7cc7c9269823e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:02 UTC) from {username='slave'} at 127.0.0.1:41362
I20260808 22:54:02.545238 18466 tablet_replica.cc:330] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: stopping tablet replica
I20260808 22:54:02.545329 18466 raft_consensus.cc:2229] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:02.545411 18466 raft_consensus.cc:2258] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:02.545454 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 17771
I20260808 22:54:02.546032 18467 tablet_replica.cc:330] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: stopping tablet replica
I20260808 22:54:02.546111 18467 raft_consensus.cc:2229] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:54:02.546182 18467 raft_consensus.cc:2258] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:02.548383 18467 ts_tablet_manager.cc:1905] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:02.550580 18467 ts_tablet_manager.cc:1918] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:02.550658 18467 log.cc:1196] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/91bab11c937748c49fd7cc7c9269823e
I20260808 22:54:02.550976 18467 ts_tablet_manager.cc:1939] T 91bab11c937748c49fd7cc7c9269823e P 6e2bc30ac2c1478da58fe597a7a2b70d: Deleting consensus metadata
I20260808 22:54:02.551419 17636 catalog_manager.cc:4841] TS 6e2bc30ac2c1478da58fe597a7a2b70d (127.22.30.2:35361): tablet 91bab11c937748c49fd7cc7c9269823e (table test-table [id=d41a3957ac044ec2a0145c1865fe2e10]) successfully deleted
I20260808 22:54:02.552397 18466 ts_tablet_manager.cc:1905] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:02.553846 18466 ts_tablet_manager.cc:1918] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:02.553908 18466 log.cc:1196] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/91bab11c937748c49fd7cc7c9269823e
I20260808 22:54:02.554137 18466 ts_tablet_manager.cc:1939] T 91bab11c937748c49fd7cc7c9269823e P 340ea44366be4c1490249b1d8758d9a7: Deleting consensus metadata
I20260808 22:54:02.554518 17634 catalog_manager.cc:4841] TS 340ea44366be4c1490249b1d8758d9a7 (127.22.30.3:45315): tablet 91bab11c937748c49fd7cc7c9269823e (table test-table [id=d41a3957ac044ec2a0145c1865fe2e10]) successfully deleted
W20260808 22:54:02.558104 17638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:35917 (error 108)
I20260808 22:54:02.558552 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:02.558425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:35917 (local address 127.0.0.1:34092)
0808 22:54:02.558477 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:02.558516 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:54:02.558647 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 17980
W20260808 22:54:02.558748 17638 catalog_manager.cc:4584] TS 5b12d2e750f24aae88a6691e89dbcd1b (127.22.30.1:35917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91bab11c937748c49fd7cc7c9269823e: Network error: Client connection negotiation failed: client connection to 127.22.30.1:35917: connect: Connection refused (error 111)
I20260808 22:54:02.567317 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 18199
I20260808 22:54:02.578577 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 17605
2026-08-08T22:54:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (831 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:02 dist-test-slave-x4qh krb5kdc[18473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:02 dist-test-slave-x4qh krb5kdc[18473](info): set up 2 sockets
Aug 08 22:54:02 dist-test-slave-x4qh krb5kdc[18473](info): commencing operation
krb5kdc: starting...
W20260808 22:54:03.755824 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:03 dist-test-slave-x4qh krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:03.867978 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39681
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:39681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:03.946314 18496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:03.946527 18496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:03.946563 18496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:03.947665 18496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:03.947726 18496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:03.947749 18496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:03.947769 18496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:03.947803 18496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:03.949178 18496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:39681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:03.949429 18496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:03.949616 18496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:03.954128 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:03.954169 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:03.954171 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:03.954196 18496 server_base.cc:1034] running on GCE node
I20260808 22:54:03.954523 18496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:03.954835 18496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:03.955986 18496 hybrid_clock.cc:648] HybridClock initialized: now 1786229643955958 us; error 45 us; skew 500 ppm
Aug 08 22:54:03 dist-test-slave-x4qh krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229643, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:03.958274 18496 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:03.958705 18496 webserver.cc:460] Webserver started at http://127.22.30.62:42989/ using document root <none> and password file <none>
I20260808 22:54:03.958882 18496 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:03.958921 18496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:03.959002 18496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:03.959960 18496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "596fc3c03d5e4a6fa9eef984d0640d2d"
format_stamp: "Formatted at 2026-08-08 22:54:03 on dist-test-slave-x4qh"
server_key: "abbe532d7c29c8ef9ae698a621de19db"
server_key_iv: "b0ebef9749891b1de3201783589c3be0"
server_key_version: "encryptionkey@0"
I20260808 22:54:03.960242 18496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "596fc3c03d5e4a6fa9eef984d0640d2d"
format_stamp: "Formatted at 2026-08-08 22:54:03 on dist-test-slave-x4qh"
server_key: "abbe532d7c29c8ef9ae698a621de19db"
server_key_iv: "b0ebef9749891b1de3201783589c3be0"
server_key_version: "encryptionkey@0"
I20260808 22:54:03.961376 18496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:03.962069 18518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:03.962240 18496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:03.962312 18496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "596fc3c03d5e4a6fa9eef984d0640d2d"
format_stamp: "Formatted at 2026-08-08 22:54:03 on dist-test-slave-x4qh"
server_key: "abbe532d7c29c8ef9ae698a621de19db"
server_key_iv: "b0ebef9749891b1de3201783589c3be0"
server_key_version: "encryptionkey@0"
I20260808 22:54:03.962357 18496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:03.980176 18496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:03.981624 18496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:03.981714 18496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:03.985365 18496 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:39681
I20260808 22:54:03.985410 18594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:39681 every 8 connection(s)
I20260808 22:54:03.985755 18496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:03.986573 18595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:03.988680 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: Bootstrap starting.
I20260808 22:54:03.989195 18595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:03.989477 18595 log.cc:824] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:03.990089 18595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: No bootstrap required, opened a new log
I20260808 22:54:03.991550 18595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39681 } }
I20260808 22:54:03.991685 18595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:03.991708 18595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596fc3c03d5e4a6fa9eef984d0640d2d, State: Initialized, Role: FOLLOWER
I20260808 22:54:03.991864 18595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39681 } }
I20260808 22:54:03.991945 18595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:03.991992 18595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:03.992053 18595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:03.992254 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 18496
I20260808 22:54:03.992393 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:03.992585 22648 external_mini_cluster.cc:1428] Setting key 819479075603e2c5b0ccb28c0bf433f1
I20260808 22:54:03.992755 18595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39681 } }
I20260808 22:54:03.992872 18595 leader_election.cc:302] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 596fc3c03d5e4a6fa9eef984d0640d2d; no voters:
I20260808 22:54:03.993022 18595 leader_election.cc:288] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:03.993216 18598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:03.993290 18595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:03.993402 18598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [term 1 LEADER]: Becoming Leader. State: Replica: 596fc3c03d5e4a6fa9eef984d0640d2d, State: Running, Role: LEADER
I20260808 22:54:03.993574 18598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39681 } }
I20260808 22:54:03.994006 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596fc3c03d5e4a6fa9eef984d0640d2d. Latest consensus state: current_term: 1 leader_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39681 } } }
I20260808 22:54:03.994015 18604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596fc3c03d5e4a6fa9eef984d0640d2d" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39681 } } }
I20260808 22:54:03.994182 18604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:03.994390 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d [sys.catalog]: This master's current role is: LEADER
W20260808 22:54:03.995272 18633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:03.995436 18633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:03.996223 18616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:03.996424 18616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:03.998123 18616 catalog_manager.cc:1347] Generated new cluster ID: 890f5f37c0d947a7b51e4d03844f98e4
I20260808 22:54:03.998183 18616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:03 dist-test-slave-x4qh krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:04.004135 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:03.993065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.22.30.62:39681)
0808 22:54:03.993222 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:03.993231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:03.993271 (+ 40us) server_negotiation.cc:407] Connection header received
0808 22:54:03.993390 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:03.993397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:03.993454 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:03.994443 (+ 989us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:03.994557 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:03.995891 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:03.996945 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:03.997543 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:03.997663 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.000361 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.000365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.000366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:04.000376 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.002567 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.002949 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.002952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.002952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.003058 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.003261 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.003262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.003262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.003525 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.003671 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.003796 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.003927 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260808 22:54:04.004150 18616 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:04.004786 18616 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:04.012013 18616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: Generated new TSK 0
I20260808 22:54:04.012157 18616 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:54:04.039320 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39681
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:04.121625 18644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:04.121827 18644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:04.121878 18644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:04.122920 18644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:04.122975 18644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:04.123018 18644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:54:04.124428 18644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:04.124660 18644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:04.124853 18644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:04.130050 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.130039 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.130025 18653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.130227 18644 server_base.cc:1034] running on GCE node
I20260808 22:54:04.130419 18644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:04.130648 18644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:04.131794 18644 hybrid_clock.cc:648] HybridClock initialized: now 1786229644131771 us; error 39 us; skew 500 ppm
Aug 08 22:54:04 dist-test-slave-x4qh krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229644, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:04.134181 18644 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:54:04.134610 18644 webserver.cc:460] Webserver started at http://127.22.30.1:38565/ using document root <none> and password file <none>
I20260808 22:54:04.134833 18644 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:04.134912 18644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:04.135043 18644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:04.136070 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "b74988cff25b426595733870e0461474"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "50fd0f4e70034bcc9fdcb506a48f1f61"
server_key_iv: "60e345f535b4d62a93995aa44ec92ed4"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.136408 18644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "b74988cff25b426595733870e0461474"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "50fd0f4e70034bcc9fdcb506a48f1f61"
server_key_iv: "60e345f535b4d62a93995aa44ec92ed4"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.137603 18644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:54:04.138306 18662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.138474 18644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:04.138561 18644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "b74988cff25b426595733870e0461474"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "50fd0f4e70034bcc9fdcb506a48f1f61"
server_key_iv: "60e345f535b4d62a93995aa44ec92ed4"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.138628 18644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:04.172063 18644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:04.173884 18644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:04.174149 18644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:04.174355 18644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:04.174644 18644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:04.174696 18644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.174727 18644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:04.174763 18644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.179912 18644 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:36031
I20260808 22:54:04.179956 18847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:36031 every 8 connection(s)
I20260808 22:54:04.180215 18644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:54:04.184388 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 18644
I20260808 22:54:04.184515 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:54:04.184726 22648 external_mini_cluster.cc:1428] Setting key 7ad725645a2961e6b5f69f2c8ea5354b
Aug 08 22:54:04 dist-test-slave-x4qh krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229644, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:04.188911 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.180837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:59275 (local address 127.22.30.62:39681)
0808 22:54:04.180891 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.180893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.181080 (+ 187us) server_negotiation.cc:407] Connection header received
0808 22:54:04.181540 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.181542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.181555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.181620 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:04.182446 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.182879 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.183743 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.183943 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.184017 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.186174 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.186176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.186176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:04.186184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.187621 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.188060 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.188060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.188061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.188106 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.188373 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.188374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.188374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.188459 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.188504 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.188792 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.188854 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:54:04.189208 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.180813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:39681 (local address 127.22.30.1:59275)
0808 22:54:04.180972 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.181002 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.181447 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.181681 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.181684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.182188 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:04.182358 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.182362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.182961 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.182962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.183661 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.183663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.184031 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.184032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.184136 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.184584 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:04.184593 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:04.186091 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:04.187690 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:04.187697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:04.187699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:04.187991 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:04.188171 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:04.188172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:04.188172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:04.188313 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:04.188515 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:04.188522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:04.188665 (+ 143us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.188759 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.188915 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260808 22:54:04.189708 18849 heartbeater.cc:347] Connected to a master server at 127.22.30.62:39681
I20260808 22:54:04.189800 18849 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:04.190013 18849 heartbeater.cc:511] Master 127.22.30.62:39681 requested a full tablet report, sending...
I20260808 22:54:04.190492 18546 ts_manager.cc:194] Registered new tserver with Master: b74988cff25b426595733870e0461474 (127.22.30.1:36031)
I20260808 22:54:04.191045 18546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:59275
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:54:04.207887 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39681
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:04.281455 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:04.281636 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:04.281668 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:04.282754 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:04.282820 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:04.282862 18857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:54:04.284279 18857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:04.284539 18857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:04.284757 18857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:04.289462 18872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.289439 18869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.289469 18857 server_base.cc:1034] running on GCE node
W20260808 22:54:04.289441 18867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.289830 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:04.290037 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:04.291167 18857 hybrid_clock.cc:648] HybridClock initialized: now 1786229644291143 us; error 37 us; skew 500 ppm
Aug 08 22:54:04 dist-test-slave-x4qh krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229644, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:04.293326 18857 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:54:04.293776 18857 webserver.cc:460] Webserver started at http://127.22.30.2:43485/ using document root <none> and password file <none>
I20260808 22:54:04.294015 18857 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:04.294078 18857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:04.294203 18857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:04.295228 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "7a71febd636e48049ca6d8d2a1175fc0"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "44737fc79f9585e3514ce328fbf5b46b"
server_key_iv: "9d38960216fa7b2c356a9dc09474fe78"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.295583 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "7a71febd636e48049ca6d8d2a1175fc0"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "44737fc79f9585e3514ce328fbf5b46b"
server_key_iv: "9d38960216fa7b2c356a9dc09474fe78"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.296835 18857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:54:04.297493 18883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.297652 18857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:04.297734 18857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "7a71febd636e48049ca6d8d2a1175fc0"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "44737fc79f9585e3514ce328fbf5b46b"
server_key_iv: "9d38960216fa7b2c356a9dc09474fe78"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.297809 18857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:04.312964 18857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:04.315089 18857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:04.315236 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:04.315488 18857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:04.315788 18857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:04.315819 18857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.315866 18857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:04.315893 18857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.321244 18857 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:40315
I20260808 22:54:04.321302 19051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:40315 every 8 connection(s)
I20260808 22:54:04.321581 18857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
Aug 08 22:54:04 dist-test-slave-x4qh krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229644, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:04.329493 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.322329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:47329 (local address 127.22.30.62:39681)
0808 22:54:04.322395 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.322397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.322581 (+ 184us) server_negotiation.cc:407] Connection header received
0808 22:54:04.323034 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.323035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.323045 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.323102 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:04.324016 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.324491 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.325334 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.325505 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.325551 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.327464 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.327465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.327465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:04.327472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.328516 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.328844 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.328845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.328845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.328891 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.329045 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.329046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.329046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.329132 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.329178 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.329374 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.329435 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 22:54:04.329645 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.322297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:39681 (local address 127.22.30.2:47329)
0808 22:54:04.322460 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.322499 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.322949 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.323150 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.323153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.323753 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:04.323940 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.323944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.324564 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.324564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.325270 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.325272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.325576 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.325576 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.325664 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.326076 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:04.326083 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:04.327329 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:04.328578 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:04.328583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:04.328585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:04.328787 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:04.328932 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:04.328932 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:04.328932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:04.329005 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:04.329182 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:04.329187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:04.329281 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.329352 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.329454 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260808 22:54:04.330049 19053 heartbeater.cc:347] Connected to a master server at 127.22.30.62:39681
I20260808 22:54:04.330140 19053 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:04.330300 19053 heartbeater.cc:511] Master 127.22.30.62:39681 requested a full tablet report, sending...
I20260808 22:54:04.330618 18546 ts_manager.cc:194] Registered new tserver with Master: 7a71febd636e48049ca6d8d2a1175fc0 (127.22.30.2:40315)
I20260808 22:54:04.330955 18546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:47329
I20260808 22:54:04.331744 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 18857
I20260808 22:54:04.331873 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:54:04.332036 22648 external_mini_cluster.cc:1428] Setting key 6e5955edb5bfafc97b66c902d1df9e41
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:54:04.356063 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39681
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:04.430721 19061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:04.430888 19061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:04.430907 19061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:04.431831 19061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:04.431861 19061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:04.431885 19061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:54:04.433089 19061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:39681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:04.433245 19061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:04.433390 19061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:04.438051 19069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.438081 19068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.438169 19072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.438084 19061 server_base.cc:1034] running on GCE node
I20260808 22:54:04.438436 19061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:04.438664 19061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:04.439786 19061 hybrid_clock.cc:648] HybridClock initialized: now 1786229644439777 us; error 24 us; skew 500 ppm
Aug 08 22:54:04 dist-test-slave-x4qh krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229644, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:04.442014 19061 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:54:04.442404 19061 webserver.cc:460] Webserver started at http://127.22.30.3:39881/ using document root <none> and password file <none>
I20260808 22:54:04.442577 19061 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:04.442612 19061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:04.442689 19061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:04.443615 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "4627987b0fb041c3b8780880469d2274"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "584c6aa6faf1fae6a9031d2a70e4d02f"
server_key_iv: "9f7524b4332c3fb6adf2da4a32b0587e"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.443895 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "4627987b0fb041c3b8780880469d2274"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "584c6aa6faf1fae6a9031d2a70e4d02f"
server_key_iv: "9f7524b4332c3fb6adf2da4a32b0587e"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.445082 19061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:04.445701 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.445901 19061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:04.446000 19061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "4627987b0fb041c3b8780880469d2274"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "584c6aa6faf1fae6a9031d2a70e4d02f"
server_key_iv: "9f7524b4332c3fb6adf2da4a32b0587e"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.446069 19061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:04.453810 19061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:04.455348 19061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:04.455519 19061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:04.455701 19061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:04.455996 19061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:04.456028 19061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.456064 19061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:04.456090 19061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.461324 19061 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:34785
I20260808 22:54:04.461386 19252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:34785 every 8 connection(s)
I20260808 22:54:04.461630 19061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
Aug 08 22:54:04 dist-test-slave-x4qh krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229644, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:04.469660 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.462218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:46655 (local address 127.22.30.62:39681)
0808 22:54:04.462316 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.462318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.462464 (+ 146us) server_negotiation.cc:407] Connection header received
0808 22:54:04.462875 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.462876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.462886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.462927 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:04.463859 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.464305 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.465250 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.465436 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.465487 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.467473 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.467474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.467475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:04.467481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.468689 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.468997 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.468998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.468998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.469042 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.469221 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.469221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.469222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.469311 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.469356 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.469554 (+ 198us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.469605 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 22:54:04.469833 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.462188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:39681 (local address 127.22.30.3:46655)
0808 22:54:04.462366 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.462397 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.462772 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.462973 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.462976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.463610 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:04.463787 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.463791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.464388 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.464389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.465184 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.465186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.465509 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.465510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.465623 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.466071 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:04.466078 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:04.467333 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:04.468749 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:04.468754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:04.468755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:04.468941 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:04.469093 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:04.469093 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:04.469094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:04.469175 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:04.469358 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:04.469362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:04.469454 (+ 92us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.469529 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.469626 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260808 22:54:04.470257 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 19061
I20260808 22:54:04.470301 19253 heartbeater.cc:347] Connected to a master server at 127.22.30.62:39681
I20260808 22:54:04.470369 19253 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:04.470397 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:54:04.470536 19253 heartbeater.cc:511] Master 127.22.30.62:39681 requested a full tablet report, sending...
I20260808 22:54:04.470559 22648 external_mini_cluster.cc:1428] Setting key 7266408cd0dbd0cc832937005acefa05
I20260808 22:54:04.470880 18546 ts_manager.cc:194] Registered new tserver with Master: 4627987b0fb041c3b8780880469d2274 (127.22.30.3:34785)
I20260808 22:54:04.471210 18546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:46655
I20260808 22:54:04.471387 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:54:04.477869 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.472752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.22.30.62:39681)
0808 22:54:04.472802 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.472804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.472909 (+ 105us) server_negotiation.cc:407] Connection header received
0808 22:54:04.472956 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.472956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.472962 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.473002 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:04.473462 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.473846 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.474512 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.474671 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.474725 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.475828 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.475830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.475830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:04.475835 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.477079 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.477368 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.477369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.477369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.477427 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:04.477558 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:04.477558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:04.477558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:04.477648 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.477713 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.477777 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.477819 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:54:04.479218 18546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:54:04.480578 18546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:54:04.485896 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.482321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:34785 (local address 127.0.0.1:56322)
0808 22:54:04.482461 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.482492 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.482565 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.482883 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.482886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.482888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.482957 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.482959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.483860 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.483861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.484742 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.484745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.485607 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.485608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.485679 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.485691 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.485760 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.485826 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260808 22:54:04.485904 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.482500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.22.30.3:34785)
0808 22:54:04.482745 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.482748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.482755 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:04.482781 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.482782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.482792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.482875 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.483015 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.483770 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.484830 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.485536 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.485606 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.485651 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.485776 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.485830 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260808 22:54:04.486179 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.482321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40315 (local address 127.0.0.1:38454)
0808 22:54:04.482645 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.482655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.482696 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.482929 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.482929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.482931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.482969 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.482971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.484210 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.484211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.485145 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.485147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.486015 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.486016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.486082 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.486085 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.486108 (+ 23us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.486135 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":4,"spinlock_wait_cycles":384,"thread_start_us":57,"threads_started":1}
I20260808 22:54:04.486418 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.482545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.22.30.2:40315)
0808 22:54:04.482714 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.482718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.482728 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:54:04.482766 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.482768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.482782 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.482853 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.483036 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.484103 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.485236 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.485910 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.486228 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.486277 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.486297 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.486325 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260808 22:54:04.486615 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.482330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36031 (local address 127.0.0.1:38722)
0808 22:54:04.482778 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.482786 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.482820 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.483333 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.483335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.483337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.483441 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.483442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.484623 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.484624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.485472 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.485475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.486379 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.486380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.486445 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.486449 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.486506 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.486549 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":96,"threads_started":1}
I20260808 22:54:04.486828 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.482583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.22.30.1:36031)
0808 22:54:04.483117 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.483120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.483129 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:04.483162 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.483164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.483176 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.483242 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.483518 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.484526 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.485549 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.486290 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.486370 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.486426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.486708 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.486752 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":74,"threads_started":1}
I20260808 22:54:04.487078 19153 tablet_service.cc:1467] Processing CreateTablet for tablet d283c44cd11c4ae3bce05a8b9aff8cb2 (DEFAULT_TABLE table=test-table [id=f39252000b3b4a97886f7a0e03e12f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:04.487107 18953 tablet_service.cc:1467] Processing CreateTablet for tablet d283c44cd11c4ae3bce05a8b9aff8cb2 (DEFAULT_TABLE table=test-table [id=f39252000b3b4a97886f7a0e03e12f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:04.487447 18953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d283c44cd11c4ae3bce05a8b9aff8cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:04.487448 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d283c44cd11c4ae3bce05a8b9aff8cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:04.487494 18740 tablet_service.cc:1467] Processing CreateTablet for tablet d283c44cd11c4ae3bce05a8b9aff8cb2 (DEFAULT_TABLE table=test-table [id=f39252000b3b4a97886f7a0e03e12f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:04.487757 18740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d283c44cd11c4ae3bce05a8b9aff8cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:04.489642 19268 tablet_bootstrap.cc:492] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Bootstrap starting.
I20260808 22:54:04.490247 19268 tablet_bootstrap.cc:654] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:04.490800 19269 tablet_bootstrap.cc:492] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: Bootstrap starting.
I20260808 22:54:04.490891 19268 log.cc:824] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:04.491237 19269 tablet_bootstrap.cc:654] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:04.491499 19270 tablet_bootstrap.cc:492] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: Bootstrap starting.
I20260808 22:54:04.491691 19268 tablet_bootstrap.cc:492] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: No bootstrap required, opened a new log
I20260808 22:54:04.491763 19268 ts_tablet_manager.cc:1397] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:04.491788 19269 log.cc:824] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:04.492056 19270 tablet_bootstrap.cc:654] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:04.492512 19270 log.cc:824] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:04.492524 19269 tablet_bootstrap.cc:492] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: No bootstrap required, opened a new log
I20260808 22:54:04.492579 19269 ts_tablet_manager.cc:1397] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:04.493161 19270 tablet_bootstrap.cc:492] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: No bootstrap required, opened a new log
I20260808 22:54:04.493234 19270 ts_tablet_manager.cc:1397] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:04.493734 19268 raft_consensus.cc:348] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.493903 19268 raft_consensus.cc:374] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:04.493937 19268 raft_consensus.cc:729] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4627987b0fb041c3b8780880469d2274, State: Initialized, Role: FOLLOWER
I20260808 22:54:04.494086 19268 consensus_queue.cc:260] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.494364 19268 ts_tablet_manager.cc:1428] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 22:54:04.494478 19253 heartbeater.cc:503] Master 127.22.30.62:39681 was elected leader, sending a full tablet report...
I20260808 22:54:04.494478 19269 raft_consensus.cc:348] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.494664 19269 raft_consensus.cc:374] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:04.494712 19269 raft_consensus.cc:729] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a71febd636e48049ca6d8d2a1175fc0, State: Initialized, Role: FOLLOWER
I20260808 22:54:04.494832 19269 consensus_queue.cc:260] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.495108 19053 heartbeater.cc:503] Master 127.22.30.62:39681 was elected leader, sending a full tablet report...
I20260808 22:54:04.495312 19269 ts_tablet_manager.cc:1428] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:04.495199 19270 raft_consensus.cc:348] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.495393 19270 raft_consensus.cc:374] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:04.495445 19270 raft_consensus.cc:729] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b74988cff25b426595733870e0461474, State: Initialized, Role: FOLLOWER
I20260808 22:54:04.495560 19270 consensus_queue.cc:260] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.495882 19270 ts_tablet_manager.cc:1428] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:04.495959 18849 heartbeater.cc:503] Master 127.22.30.62:39681 was elected leader, sending a full tablet report...
I20260808 22:54:04.566342 19278 raft_consensus.cc:482] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:04.566473 19278 raft_consensus.cc:504] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.566844 19278 leader_election.cc:288] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4627987b0fb041c3b8780880469d2274 (127.22.30.3:34785), b74988cff25b426595733870e0461474 (127.22.30.1:36031)
I20260808 22:54:04.567538 19277 raft_consensus.cc:482] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:04.567655 19277 raft_consensus.cc:504] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.568007 19277 leader_election.cc:288] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a71febd636e48049ca6d8d2a1175fc0 (127.22.30.2:40315), b74988cff25b426595733870e0461474 (127.22.30.1:36031)
I20260808 22:54:04.570706 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.567099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:50691 (local address 127.22.30.1:36031)
0808 22:54:04.567160 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.567163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.567294 (+ 131us) server_negotiation.cc:407] Connection header received
0808 22:54:04.567325 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.567327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.567339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.567445 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.567640 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.568334 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.569837 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.570372 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.570413 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.570424 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.570619 (+ 195us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.570650 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:54:04.570703 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.566999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:34785 (local address 127.22.30.2:44389)
0808 22:54:04.567053 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.567064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.567105 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.567317 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.567318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.567320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.567417 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.567419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.568514 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.568516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.569732 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.569734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.570465 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.570466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.570543 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.570545 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.570589 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.570631 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":6}
I20260808 22:54:04.570981 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.567090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:44389 (local address 127.22.30.3:34785)
0808 22:54:04.567178 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.567180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.567189 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:04.567217 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.567218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.567231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.567302 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.567514 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.568265 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.569837 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.570364 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.570426 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.570435 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.570887 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.570930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 22:54:04.571218 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "7a71febd636e48049ca6d8d2a1175fc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4627987b0fb041c3b8780880469d2274" is_pre_election: true
I20260808 22:54:04.571317 19184 raft_consensus.cc:2454] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a71febd636e48049ca6d8d2a1175fc0 in term 0.
I20260808 22:54:04.571452 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.567015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36031 (local address 127.22.30.2:50691)
0808 22:54:04.567212 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.567221 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.567272 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.567474 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.567475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.567477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.567558 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.567560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.568417 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.568418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.569732 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.569734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.570477 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.570477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.570538 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.570541 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.571161 (+ 620us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.571206 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":125,"thread_start_us":126,"threads_started":1}
I20260808 22:54:04.571697 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.568458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:40585 (local address 127.22.30.2:40315)
0808 22:54:04.568500 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.568502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.568509 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:04.568525 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.568526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.568534 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.568578 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.568767 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.569441 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.570833 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.571332 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.571404 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.571412 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.571613 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.571642 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":23}
I20260808 22:54:04.571702 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.568167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36031 (local address 127.22.30.3:52065)
0808 22:54:04.568347 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.568355 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.568387 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.568706 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.568708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.568709 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.568765 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.568766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.569627 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.569627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.570718 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.570720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.571503 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.571503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.571568 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.571570 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.571598 (+ 28us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.571641 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":8,"thread_start_us":82,"threads_started":1}
I20260808 22:54:04.571554 18890 leader_election.cc:302] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4627987b0fb041c3b8780880469d2274, 7a71febd636e48049ca6d8d2a1175fc0; no voters:
I20260808 22:54:04.571883 19284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.568108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40315 (local address 127.22.30.3:40585)
0808 22:54:04.568318 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.568328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.568380 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.568604 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.568605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.568607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:04.568681 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.568683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.569527 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.569527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.570718 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.570720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.571472 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.571473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.571536 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.571539 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.571786 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.571825 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":26}
I20260808 22:54:04.571893 19278 raft_consensus.cc:2781] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:04.571946 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.568432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:52065 (local address 127.22.30.1:36031)
0808 22:54:04.568585 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.568587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.568594 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:04.568615 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.568616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.568624 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.568688 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:04.568841 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.569541 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.570820 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.571426 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.571808 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.571818 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.571835 (+ 17us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.571866 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260808 22:54:04.571954 19278 raft_consensus.cc:482] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:04.571981 19278 raft_consensus.cc:3037] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:04.572121 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "7a71febd636e48049ca6d8d2a1175fc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b74988cff25b426595733870e0461474" is_pre_election: true
W20260808 22:54:04.572187 19054 tablet.cc:2369] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:54:04.572206 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "4627987b0fb041c3b8780880469d2274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b74988cff25b426595733870e0461474" is_pre_election: true
I20260808 22:54:04.572240 18775 raft_consensus.cc:2454] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a71febd636e48049ca6d8d2a1175fc0 in term 0.
I20260808 22:54:04.572289 18773 raft_consensus.cc:2454] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4627987b0fb041c3b8780880469d2274 in term 0.
I20260808 22:54:04.572481 19088 leader_election.cc:302] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4627987b0fb041c3b8780880469d2274, b74988cff25b426595733870e0461474; no voters:
I20260808 22:54:04.572602 19277 raft_consensus.cc:2781] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:04.572685 19277 raft_consensus.cc:482] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:04.572707 19277 raft_consensus.cc:3037] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:04.572697 18986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "4627987b0fb041c3b8780880469d2274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a71febd636e48049ca6d8d2a1175fc0" is_pre_election: true
I20260808 22:54:04.572904 19278 raft_consensus.cc:504] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.573038 18986 raft_consensus.cc:2379] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4627987b0fb041c3b8780880469d2274 in current term 1: Already voted for candidate 7a71febd636e48049ca6d8d2a1175fc0 in this term.
I20260808 22:54:04.573170 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "7a71febd636e48049ca6d8d2a1175fc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b74988cff25b426595733870e0461474"
I20260808 22:54:04.573196 19184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "7a71febd636e48049ca6d8d2a1175fc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4627987b0fb041c3b8780880469d2274"
I20260808 22:54:04.573239 18773 raft_consensus.cc:3037] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:04.573531 19277 raft_consensus.cc:504] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.573046 19278 leader_election.cc:288] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [CANDIDATE]: Term 1 election: Requested vote from peers 4627987b0fb041c3b8780880469d2274 (127.22.30.3:34785), b74988cff25b426595733870e0461474 (127.22.30.1:36031)
I20260808 22:54:04.573660 19277 leader_election.cc:288] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [CANDIDATE]: Term 1 election: Requested vote from peers 7a71febd636e48049ca6d8d2a1175fc0 (127.22.30.2:40315), b74988cff25b426595733870e0461474 (127.22.30.1:36031)
I20260808 22:54:04.573801 19184 raft_consensus.cc:2379] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7a71febd636e48049ca6d8d2a1175fc0 in current term 1: Already voted for candidate 4627987b0fb041c3b8780880469d2274 in this term.
I20260808 22:54:04.573836 18986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "4627987b0fb041c3b8780880469d2274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a71febd636e48049ca6d8d2a1175fc0"
I20260808 22:54:04.573928 18986 raft_consensus.cc:2379] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4627987b0fb041c3b8780880469d2274 in current term 1: Already voted for candidate 7a71febd636e48049ca6d8d2a1175fc0 in this term.
I20260808 22:54:04.574062 18775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d283c44cd11c4ae3bce05a8b9aff8cb2" candidate_uuid: "4627987b0fb041c3b8780880469d2274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b74988cff25b426595733870e0461474"
I20260808 22:54:04.574180 18773 raft_consensus.cc:2454] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a71febd636e48049ca6d8d2a1175fc0 in term 1.
I20260808 22:54:04.574304 19088 leader_election.cc:302] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4627987b0fb041c3b8780880469d2274; no voters: 7a71febd636e48049ca6d8d2a1175fc0, b74988cff25b426595733870e0461474
I20260808 22:54:04.574331 18888 leader_election.cc:302] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a71febd636e48049ca6d8d2a1175fc0, b74988cff25b426595733870e0461474; no voters: 4627987b0fb041c3b8780880469d2274
I20260808 22:54:04.574399 19277 raft_consensus.cc:2726] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 22:54:04.574451 19278 raft_consensus.cc:2781] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:04.574585 19278 raft_consensus.cc:686] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a71febd636e48049ca6d8d2a1175fc0, State: Running, Role: LEADER
I20260808 22:54:04.574680 19278 consensus_queue.cc:237] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } }
I20260808 22:54:04.575496 18546 catalog_manager.cc:5495] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a71febd636e48049ca6d8d2a1175fc0 (127.22.30.2). New cstate: current_term: 1 leader_uuid: "7a71febd636e48049ca6d8d2a1175fc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a71febd636e48049ca6d8d2a1175fc0" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 } health_report { overall_health: UNKNOWN } } }
I20260808 22:54:04.600593 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.598300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.22.30.2:40315)
0808 22:54:04.598354 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.598356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.598364 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:04.598397 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.598398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.598413 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.598478 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:04.598604 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.599062 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.599831 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.600017 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.600083 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.600249 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:04.600339 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:04.600438 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.600495 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.600504 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:54:04.602191 19184 raft_consensus.cc:1261] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 1 FOLLOWER]: Refusing update from remote peer 7a71febd636e48049ca6d8d2a1175fc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:04.602202 18773 raft_consensus.cc:1261] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 1 FOLLOWER]: Refusing update from remote peer 7a71febd636e48049ca6d8d2a1175fc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:04.602491 19278 consensus_queue.cc:1035] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b74988cff25b426595733870e0461474" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:04.602615 19288 consensus_queue.cc:1035] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4627987b0fb041c3b8780880469d2274" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:04.604951 19292 mvcc.cc:205] Tried to move back new op lower bound from 7316396624288219136 to 7316396624178368512. Current Snapshot: MvccSnapshot[applied={T|T < 7316396624288219136}]
I20260808 22:54:04.607782 18543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:04.607879 18543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:04.608596 18543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 596fc3c03d5e4a6fa9eef984d0640d2d: Sending DeleteTablet for 3 replicas of tablet d283c44cd11c4ae3bce05a8b9aff8cb2
I20260808 22:54:04.608924 18740 tablet_service.cc:1514] Processing DeleteTablet for tablet d283c44cd11c4ae3bce05a8b9aff8cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:04 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:38722
I20260808 22:54:04.608942 18953 tablet_service.cc:1514] Processing DeleteTablet for tablet d283c44cd11c4ae3bce05a8b9aff8cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:04 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:38454
I20260808 22:54:04.609184 19307 tablet_replica.cc:330] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474: stopping tablet replica
I20260808 22:54:04.609225 19153 tablet_service.cc:1514] Processing DeleteTablet for tablet d283c44cd11c4ae3bce05a8b9aff8cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:04 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:56322
I20260808 22:54:04.609386 19307 raft_consensus.cc:2229] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:04.609500 19307 raft_consensus.cc:2258] T d283c44cd11c4ae3bce05a8b9aff8cb2 P b74988cff25b426595733870e0461474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:04.609491 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 18644
I20260808 22:54:04.612823 19309 tablet_replica.cc:330] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: stopping tablet replica
I20260808 22:54:04.612929 19309 raft_consensus.cc:2229] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:04.613029 19309 raft_consensus.cc:2258] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:04.613452 19309 ts_tablet_manager.cc:1905] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:04.615597 19309 ts_tablet_manager.cc:1918] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:04.615686 19309 log.cc:1196] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/d283c44cd11c4ae3bce05a8b9aff8cb2
I20260808 22:54:04.616046 19309 ts_tablet_manager.cc:1939] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 4627987b0fb041c3b8780880469d2274: Deleting consensus metadata
I20260808 22:54:04.616760 18523 catalog_manager.cc:4841] TS 4627987b0fb041c3b8780880469d2274 (127.22.30.3:34785): tablet d283c44cd11c4ae3bce05a8b9aff8cb2 (table test-table [id=f39252000b3b4a97886f7a0e03e12f21]) successfully deleted
W20260808 22:54:04.621198 18521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:36031 (error 108)
I20260808 22:54:04.621340 19308 tablet_replica.cc:330] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0: stopping tablet replica
I20260808 22:54:04.621493 19308 raft_consensus.cc:2229] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:54:04.621610 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.621488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:36031 (local address 127.0.0.1:38724)
0808 22:54:04.621541 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.621575 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 22:54:04.621718 19308 raft_consensus.cc:2258] T d283c44cd11c4ae3bce05a8b9aff8cb2 P 7a71febd636e48049ca6d8d2a1175fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 22:54:04.621723 18521 catalog_manager.cc:4584] TS b74988cff25b426595733870e0461474 (127.22.30.1:36031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d283c44cd11c4ae3bce05a8b9aff8cb2: Network error: Client connection negotiation failed: client connection to 127.22.30.1:36031: connect: Connection refused (error 111)
I20260808 22:54:04.621858 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 18857
I20260808 22:54:04.638188 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 19061
I20260808 22:54:04.638283 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.638137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40315 (local address 127.0.0.1:38470)
0808 22:54:04.638198 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.638226 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.2:40315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 22:54:04.646332 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 18496
2026-08-08T22:54:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T22:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:04Z Disabled control of system clock
I20260808 22:54:04.680193 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:35565
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:35565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:04.756718 19317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:04.756910 19317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:04.756930 19317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:04.757928 19317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:04.757958 19317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:04.757970 19317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:04.757982 19317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:04.757992 19317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:04.759110 19317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:35565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:35565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:04.759286 19317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:04.759510 19317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:04.763679 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.763794 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.763846 19317 server_base.cc:1034] running on GCE node
W20260808 22:54:04.763679 19323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.764120 19317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:04.764401 19317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:04.765539 19317 hybrid_clock.cc:648] HybridClock initialized: now 1786229644765523 us; error 36 us; skew 500 ppm
I20260808 22:54:04.771977 19317 webserver.cc:460] Webserver started at http://127.22.30.62:38481/ using document root <none> and password file <none>
I20260808 22:54:04.772217 19317 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:04.772260 19317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:04.772413 19317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:04.773334 19317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "731e8ddcf92b4d67a846beba0b303e07"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "cd13fb8143d4bf341614ba8d59aec2ae"
server_key_iv: "b4257b7d3203c9ac1d090dc8a0c797e8"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.773653 19317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "731e8ddcf92b4d67a846beba0b303e07"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "cd13fb8143d4bf341614ba8d59aec2ae"
server_key_iv: "b4257b7d3203c9ac1d090dc8a0c797e8"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.774902 19317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:04.775602 19338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.775756 19317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:04.775839 19317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "731e8ddcf92b4d67a846beba0b303e07"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "cd13fb8143d4bf341614ba8d59aec2ae"
server_key_iv: "b4257b7d3203c9ac1d090dc8a0c797e8"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.775890 19317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:04.786170 19317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:04.786381 19317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:04.786473 19317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:04.789955 19317 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:35565
I20260808 22:54:04.789999 19428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:35565 every 8 connection(s)
I20260808 22:54:04.790246 19317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:04.791054 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:04.792991 19430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07: Bootstrap starting.
I20260808 22:54:04.793507 19430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:04.793787 19430 log.cc:824] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:04.794148 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 19317
I20260808 22:54:04.794270 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:04.794464 22648 external_mini_cluster.cc:1428] Setting key e739d1ab69fe951e3c3e90a77384e884
I20260808 22:54:04.794462 19430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07: No bootstrap required, opened a new log
I20260808 22:54:04.797246 19430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731e8ddcf92b4d67a846beba0b303e07" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35565 } }
I20260808 22:54:04.797396 19430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:04.797421 19430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731e8ddcf92b4d67a846beba0b303e07, State: Initialized, Role: FOLLOWER
I20260808 22:54:04.797520 19430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731e8ddcf92b4d67a846beba0b303e07" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35565 } }
I20260808 22:54:04.797601 19430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:04.797631 19430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:04.797675 19430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:04.798158 19430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731e8ddcf92b4d67a846beba0b303e07" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35565 } }
I20260808 22:54:04.798278 19430 leader_election.cc:302] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 731e8ddcf92b4d67a846beba0b303e07; no voters:
I20260808 22:54:04.798422 19430 leader_election.cc:288] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:04.798481 19438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:04.798619 19430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:04.798625 19438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [term 1 LEADER]: Becoming Leader. State: Replica: 731e8ddcf92b4d67a846beba0b303e07, State: Running, Role: LEADER
I20260808 22:54:04.798732 19438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731e8ddcf92b4d67a846beba0b303e07" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35565 } }
I20260808 22:54:04.799046 19439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "731e8ddcf92b4d67a846beba0b303e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731e8ddcf92b4d67a846beba0b303e07" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35565 } } }
I20260808 22:54:04.799139 19439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:04.799049 19441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 731e8ddcf92b4d67a846beba0b303e07. Latest consensus state: current_term: 1 leader_uuid: "731e8ddcf92b4d67a846beba0b303e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731e8ddcf92b4d67a846beba0b303e07" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 35565 } } }
I20260808 22:54:04.799284 19441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:04.799695 19451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:04.800248 19451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:04.800478 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.794915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.22.30.62:35565)
0808 22:54:04.795120 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.795129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.795160 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:54:04.795754 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.795762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.795855 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.795994 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:04.796144 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.797507 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.798235 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.798812 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.799482 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.799726 (+ 244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.799730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.799730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:04.799906 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.799943 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:04.799955 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.800178 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.800211 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.800291 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260808 22:54:04.801664 19451 catalog_manager.cc:1347] Generated new cluster ID: 7afb66e8b1774652b3f6df3be101701c
I20260808 22:54:04.801723 19451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:04.818384 19451 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:04.819463 19451 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:04.823418 19451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07: Generated new TSK 0
I20260808 22:54:04.823589 19451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:04.825752 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35565
--builtin_ntp_servers=127.22.30.20:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:04.911798 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:04.912006 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:04.912040 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:04.913148 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:04.913208 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:04.913249 19469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:54:04.914796 19469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:04.915072 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:04.915390 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:04.921515 19478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.921502 19481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:04.921517 19477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:04.921826 19469 server_base.cc:1034] running on GCE node
I20260808 22:54:04.922089 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:04.922338 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:04.923485 19469 hybrid_clock.cc:648] HybridClock initialized: now 1786229644923464 us; error 42 us; skew 500 ppm
I20260808 22:54:04.924765 19469 webserver.cc:460] Webserver started at http://127.22.30.1:41357/ using document root <none> and password file <none>
I20260808 22:54:04.925036 19469 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:04.925107 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:04.925268 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:04.926337 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "d839f8661f0a489a8b0d727aded9abb5"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "7f43caa09987665f6e31d12913f9b3ae"
server_key_iv: "f7cf4ac296c784a9f3649aed3a05f61f"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.926687 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "d839f8661f0a489a8b0d727aded9abb5"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "7f43caa09987665f6e31d12913f9b3ae"
server_key_iv: "f7cf4ac296c784a9f3649aed3a05f61f"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.928046 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:04.928773 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.928982 19469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:04.929069 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "d839f8661f0a489a8b0d727aded9abb5"
format_stamp: "Formatted at 2026-08-08 22:54:04 on dist-test-slave-x4qh"
server_key: "7f43caa09987665f6e31d12913f9b3ae"
server_key_iv: "f7cf4ac296c784a9f3649aed3a05f61f"
server_key_version: "encryptionkey@0"
I20260808 22:54:04.929137 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:04.952589 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:04.952872 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:04.952968 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:04.953166 19469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:04.953409 19469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:04.953436 19469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.953456 19469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:04.953467 19469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:04.958741 19469 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:43753
I20260808 22:54:04.958837 19671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:43753 every 8 connection(s)
I20260808 22:54:04.959134 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:54:04.961851 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 19469
I20260808 22:54:04.961963 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:54:04.962137 22648 external_mini_cluster.cc:1428] Setting key 5569e08ab3ad4c75441bfb0339d39984
I20260808 22:54:04.963035 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35565
--builtin_ntp_servers=127.22.30.20:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 22:54:04.964385 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.959878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:45849 (local address 127.22.30.62:35565)
0808 22:54:04.959940 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:04.959942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:04.960198 (+ 256us) server_negotiation.cc:407] Connection header received
0808 22:54:04.960693 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:04.960694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:04.960711 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:04.960762 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:04.961443 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.961908 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.962994 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.963225 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.963300 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.964098 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:04.964099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:04.964100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:04.964105 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:04.964119 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:04.964123 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:04.964172 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:04.964263 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 22:54:04.964287 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:54:04.964594 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:04.959877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:35565 (local address 127.22.30.1:45849)
0808 22:54:04.960087 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:04.960124 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:04.960572 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:04.960840 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:04.960844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:04.960852 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:04.961339 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.961343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.962003 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.962004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.962922 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:04.962925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:04.963312 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:04.963313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:04.963471 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:04.963960 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:04.963961 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:04.963986 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:04.964007 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:04.964176 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:04.964181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:04.964184 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:04.964253 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:54:04.964375 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260808 22:54:04.965107 19672 heartbeater.cc:347] Connected to a master server at 127.22.30.62:35565
I20260808 22:54:04.965193 19672 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:04.965371 19672 heartbeater.cc:511] Master 127.22.30.62:35565 requested a full tablet report, sending...
I20260808 22:54:04.965838 19363 ts_manager.cc:194] Registered new tserver with Master: d839f8661f0a489a8b0d727aded9abb5 (127.22.30.1:43753)
I20260808 22:54:04.966446 19363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.1:45849
W20260808 22:54:05.039343 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:05.039603 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:05.039637 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:05.040656 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:05.040714 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:05.040750 19677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:54:05.041954 19677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:05.042209 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:05.042505 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:05.047273 19687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:05.047260 19689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:05.047451 19677 server_base.cc:1034] running on GCE node
W20260808 22:54:05.047564 19685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:05.047755 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:05.047958 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:05.049091 19677 hybrid_clock.cc:648] HybridClock initialized: now 1786229645049079 us; error 28 us; skew 500 ppm
I20260808 22:54:05.050163 19677 webserver.cc:460] Webserver started at http://127.22.30.2:44007/ using document root <none> and password file <none>
I20260808 22:54:05.050348 19677 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:05.050412 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:05.050509 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:05.051360 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "f9ad34d03207416ea4bec300e51ed1ee"
format_stamp: "Formatted at 2026-08-08 22:54:05 on dist-test-slave-x4qh"
server_key: "87a99fd5e36289a132dc0fa60597dff8"
server_key_iv: "8e2aed8bc6dec82a90d7e2f721b17ec1"
server_key_version: "encryptionkey@0"
I20260808 22:54:05.051698 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "f9ad34d03207416ea4bec300e51ed1ee"
format_stamp: "Formatted at 2026-08-08 22:54:05 on dist-test-slave-x4qh"
server_key: "87a99fd5e36289a132dc0fa60597dff8"
server_key_iv: "8e2aed8bc6dec82a90d7e2f721b17ec1"
server_key_version: "encryptionkey@0"
I20260808 22:54:05.052877 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:05.053530 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:05.053673 19677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:05.053755 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "f9ad34d03207416ea4bec300e51ed1ee"
format_stamp: "Formatted at 2026-08-08 22:54:05 on dist-test-slave-x4qh"
server_key: "87a99fd5e36289a132dc0fa60597dff8"
server_key_iv: "8e2aed8bc6dec82a90d7e2f721b17ec1"
server_key_version: "encryptionkey@0"
I20260808 22:54:05.053809 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:05.077105 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:05.077306 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:05.077394 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:05.077543 19677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:05.077761 19677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:05.077788 19677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:05.077805 19677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:05.077816 19677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:05.083041 19677 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:35421
I20260808 22:54:05.083091 19885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:35421 every 8 connection(s)
I20260808 22:54:05.083415 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:54:05.088181 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.084080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:59191 (local address 127.22.30.62:35565)
0808 22:54:05.084120 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.084122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.084334 (+ 212us) server_negotiation.cc:407] Connection header received
0808 22:54:05.084825 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.084826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.084839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.084880 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:05.085482 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.085972 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.086907 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.087105 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.087171 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.087900 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:05.087901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:05.087902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:05.087906 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:05.087921 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:05.087925 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:05.087970 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.088049 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.088094 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:54:05.088359 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.084059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:35565 (local address 127.22.30.2:59191)
0808 22:54:05.084233 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.084260 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.084701 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.084948 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.084952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.084961 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:05.085398 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.085402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.086066 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.086067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.086806 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.086808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.087181 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.087182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.087302 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.087782 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:05.087783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:05.087806 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:05.087824 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:05.087974 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:05.087979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:05.087982 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.088050 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.088159 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260808 22:54:05.088864 19886 heartbeater.cc:347] Connected to a master server at 127.22.30.62:35565
I20260808 22:54:05.088981 19886 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:05.089171 19886 heartbeater.cc:511] Master 127.22.30.62:35565 requested a full tablet report, sending...
I20260808 22:54:05.089181 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 19677
I20260808 22:54:05.089274 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:54:05.089461 22648 external_mini_cluster.cc:1428] Setting key ad83b5ffc948a38b18f6258c2fbdf5d2
I20260808 22:54:05.089543 19363 ts_manager.cc:194] Registered new tserver with Master: f9ad34d03207416ea4bec300e51ed1ee (127.22.30.2:35421)
I20260808 22:54:05.089921 19363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.2:59191
I20260808 22:54:05.090389 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35565
--builtin_ntp_servers=127.22.30.20:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:05.170481 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:05.170640 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:05.170660 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:05.171813 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:05.171849 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:05.171872 19893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:54:05.173116 19893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:35565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:05.173281 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:05.173486 19893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:05.177999 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:05.178009 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:05.177987 19905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:05.178184 19893 server_base.cc:1034] running on GCE node
I20260808 22:54:05.178432 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:05.178644 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:05.179764 19893 hybrid_clock.cc:648] HybridClock initialized: now 1786229645179747 us; error 32 us; skew 500 ppm
I20260808 22:54:05.181111 19893 webserver.cc:460] Webserver started at http://127.22.30.3:35141/ using document root <none> and password file <none>
I20260808 22:54:05.181352 19893 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:05.181406 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:05.181517 19893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:05.182387 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "3b1e829f89fd4614b05eae6f2e6df580"
format_stamp: "Formatted at 2026-08-08 22:54:05 on dist-test-slave-x4qh"
server_key: "d5fca1b525118a7c121fd58ca22391eb"
server_key_iv: "1bc5c34c9ead6a15bf1909fe3c913be5"
server_key_version: "encryptionkey@0"
I20260808 22:54:05.182713 19893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "3b1e829f89fd4614b05eae6f2e6df580"
format_stamp: "Formatted at 2026-08-08 22:54:05 on dist-test-slave-x4qh"
server_key: "d5fca1b525118a7c121fd58ca22391eb"
server_key_iv: "1bc5c34c9ead6a15bf1909fe3c913be5"
server_key_version: "encryptionkey@0"
I20260808 22:54:05.183954 19893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:05.184625 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:05.184800 19893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:05.184882 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "3b1e829f89fd4614b05eae6f2e6df580"
format_stamp: "Formatted at 2026-08-08 22:54:05 on dist-test-slave-x4qh"
server_key: "d5fca1b525118a7c121fd58ca22391eb"
server_key_iv: "1bc5c34c9ead6a15bf1909fe3c913be5"
server_key_version: "encryptionkey@0"
I20260808 22:54:05.184950 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:05.194944 19893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:05.195173 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:05.195298 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:05.195492 19893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:05.195762 19893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:05.195814 19893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:05.195866 19893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:05.195894 19893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:05.202009 19893 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:34975
I20260808 22:54:05.202075 20103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:34975 every 8 connection(s)
I20260808 22:54:05.202365 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:54:05.206815 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.203020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36183 (local address 127.22.30.62:35565)
0808 22:54:05.203068 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.203070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.203298 (+ 228us) server_negotiation.cc:407] Connection header received
0808 22:54:05.203759 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.203760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.203782 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.203830 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:05.204388 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.204849 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.205700 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.205907 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.205954 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.206598 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:05.206599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:05.206600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:05.206603 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:05.206617 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:05.206620 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:05.206669 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.206740 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.206760 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:54:05.207013 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.203020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:35565 (local address 127.22.30.3:36183)
0808 22:54:05.203193 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.203220 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.203680 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.203871 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.203874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.203881 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:05.204290 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.204293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.204918 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.204919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.205633 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.205635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.205973 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.205974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.206071 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.206483 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:05.206483 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:05.206507 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:05.206526 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:05.206677 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:05.206681 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:05.206683 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.206732 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.206809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260808 22:54:05.207261 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 19893
I20260808 22:54:05.207408 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:54:05.207471 20104 heartbeater.cc:347] Connected to a master server at 127.22.30.62:35565
I20260808 22:54:05.207554 20104 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:05.207608 22648 external_mini_cluster.cc:1428] Setting key ffd68b9f0f3ba0563835ffa68809bbc1
I20260808 22:54:05.207718 20104 heartbeater.cc:511] Master 127.22.30.62:35565 requested a full tablet report, sending...
I20260808 22:54:05.208055 19362 ts_manager.cc:194] Registered new tserver with Master: 3b1e829f89fd4614b05eae6f2e6df580 (127.22.30.3:34975)
I20260808 22:54:05.208619 19362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.3:36183
I20260808 22:54:05.208724 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:54:05.212468 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.210007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.22.30.62:35565)
0808 22:54:05.210044 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.210045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.210206 (+ 161us) server_negotiation.cc:407] Connection header received
0808 22:54:05.210233 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.210234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.210243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.210306 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:05.210436 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.210879 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.211600 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.211754 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.211807 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.212304 (+ 497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:05.212305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:05.212306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:05.212309 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:05.212318 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:05.212321 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:05.212383 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.212420 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.212427 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:54:05.213817 19362 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:54:05.215131 19362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:54:05.220921 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.217300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43753 (local address 127.0.0.1:46024)
0808 22:54:05.217664 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.217672 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.217723 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.218047 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.218048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.218051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:05.218086 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.218087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.219145 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.219146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.219930 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.219932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.220659 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.220732 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.220745 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.220811 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.220858 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":58,"threads_started":1}
I20260808 22:54:05.220935 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.217490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.22.30.1:43753)
0808 22:54:05.217828 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.217832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.217840 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:05.217875 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.217877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.217890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.217979 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:05.218157 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.219068 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220013 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.220512 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220637 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.220732 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.220825 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.220860 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20260808 22:54:05.221097 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.217089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:35421 (local address 127.0.0.1:41070)
0808 22:54:05.217352 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.217388 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.217453 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.217694 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.217697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.217700 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:05.217757 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.217759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.218695 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.218696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.219828 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.219831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.220848 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.220914 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.220917 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.220961 (+ 44us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.221004 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260808 22:54:05.221241 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.217180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.22.30.3:34975)
0808 22:54:05.217329 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.217332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.217520 (+ 188us) server_negotiation.cc:407] Connection header received
0808 22:54:05.217615 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.217617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.217649 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.217721 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:05.217875 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.218842 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220002 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.220670 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220716 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.220786 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.221127 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.221173 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260808 22:54:05.221269 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.217096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:34975 (local address 127.0.0.1:55972)
0808 22:54:05.217453 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.217463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.217549 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.217731 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.217731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.217733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:05.217784 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.217786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.218936 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.218937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.219918 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.219920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.220847 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.220914 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.220918 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.221104 (+ 186us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.221215 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":6,"thread_start_us":83,"threads_started":1}
I20260808 22:54:05.221395 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.217263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.22.30.2:35421)
0808 22:54:05.217426 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.217430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.217440 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:54:05.217533 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.217534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.217548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.217612 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:05.217824 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.218615 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.219920 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.220671 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.220735 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.220817 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.221256 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.221304 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260808 22:54:05.221668 19562 tablet_service.cc:1467] Processing CreateTablet for tablet c7093ef09cb34ab5ae7133724f248ac6 (DEFAULT_TABLE table=test-table [id=7b525f66a1c5479090f1232e73feeaf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:05.221983 19562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7093ef09cb34ab5ae7133724f248ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:05.221942 19780 tablet_service.cc:1467] Processing CreateTablet for tablet c7093ef09cb34ab5ae7133724f248ac6 (DEFAULT_TABLE table=test-table [id=7b525f66a1c5479090f1232e73feeaf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:05.222244 19780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7093ef09cb34ab5ae7133724f248ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:05.223650 19987 tablet_service.cc:1467] Processing CreateTablet for tablet c7093ef09cb34ab5ae7133724f248ac6 (DEFAULT_TABLE table=test-table [id=7b525f66a1c5479090f1232e73feeaf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:05.223905 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7093ef09cb34ab5ae7133724f248ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:05.225451 20121 tablet_bootstrap.cc:492] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5: Bootstrap starting.
I20260808 22:54:05.226132 20121 tablet_bootstrap.cc:654] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:05.226470 20121 log.cc:824] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:05.226752 20124 tablet_bootstrap.cc:492] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Bootstrap starting.
I20260808 22:54:05.226804 20123 tablet_bootstrap.cc:492] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Bootstrap starting.
I20260808 22:54:05.227326 20121 tablet_bootstrap.cc:492] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5: No bootstrap required, opened a new log
I20260808 22:54:05.227339 20124 tablet_bootstrap.cc:654] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:05.227442 20121 ts_tablet_manager.cc:1397] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:05.227484 20123 tablet_bootstrap.cc:654] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:05.227701 20124 log.cc:824] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:05.227813 20123 log.cc:824] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:05.228637 20123 tablet_bootstrap.cc:492] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: No bootstrap required, opened a new log
I20260808 22:54:05.228721 20123 ts_tablet_manager.cc:1397] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 22:54:05.229984 20121 raft_consensus.cc:348] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.230142 20121 raft_consensus.cc:374] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:05.230176 20121 raft_consensus.cc:729] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d839f8661f0a489a8b0d727aded9abb5, State: Initialized, Role: FOLLOWER
I20260808 22:54:05.230299 20121 consensus_queue.cc:260] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.230628 19672 heartbeater.cc:503] Master 127.22.30.62:35565 was elected leader, sending a full tablet report...
I20260808 22:54:05.230597 20121 ts_tablet_manager.cc:1428] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 22:54:05.231168 20123 raft_consensus.cc:348] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.231328 20123 raft_consensus.cc:374] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:05.231367 20123 raft_consensus.cc:729] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ad34d03207416ea4bec300e51ed1ee, State: Initialized, Role: FOLLOWER
I20260808 22:54:05.231499 20124 tablet_bootstrap.cc:492] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: No bootstrap required, opened a new log
I20260808 22:54:05.231487 20123 consensus_queue.cc:260] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.231572 20124 ts_tablet_manager.cc:1397] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 22:54:05.231699 20123 ts_tablet_manager.cc:1428] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 22:54:05.231832 19886 heartbeater.cc:503] Master 127.22.30.62:35565 was elected leader, sending a full tablet report...
I20260808 22:54:05.233651 20124 raft_consensus.cc:348] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.233776 20124 raft_consensus.cc:374] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:05.233819 20124 raft_consensus.cc:729] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1e829f89fd4614b05eae6f2e6df580, State: Initialized, Role: FOLLOWER
I20260808 22:54:05.233915 20124 consensus_queue.cc:260] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.234133 20124 ts_tablet_manager.cc:1428] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:05.234185 20104 heartbeater.cc:503] Master 127.22.30.62:35565 was elected leader, sending a full tablet report...
W20260808 22:54:05.333927 19887 tablet.cc:2369] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:54:05.355077 20131 raft_consensus.cc:482] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:05.355180 20131 raft_consensus.cc:504] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.355556 20131 leader_election.cc:288] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d839f8661f0a489a8b0d727aded9abb5 (127.22.30.1:43753), 3b1e829f89fd4614b05eae6f2e6df580 (127.22.30.3:34975)
I20260808 22:54:05.359141 20117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.355875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:49325 (local address 127.22.30.3:34975)
0808 22:54:05.355934 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.355936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.355948 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:54:05.355991 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.355992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.356002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.356053 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:05.356217 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.356774 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.358212 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.358747 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.358828 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.358837 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.359028 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.359069 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:54:05.359141 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.355807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:57225 (local address 127.22.30.1:43753)
0808 22:54:05.355867 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.355870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.356089 (+ 219us) server_negotiation.cc:407] Connection header received
0808 22:54:05.356159 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.356160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.356173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.356224 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:05.356375 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.356844 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.358212 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.358747 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.358828 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.358837 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.359028 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.359069 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:54:05.359145 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.355734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:34975 (local address 127.22.30.2:49325)
0808 22:54:05.355793 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.355805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.355854 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.356057 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.356059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.356060 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:05.356140 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.356142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.356851 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.356852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.358123 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.358125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.358843 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.358843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.358945 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.358951 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.359012 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.359060 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":74}
I20260808 22:54:05.359151 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.355741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43753 (local address 127.22.30.2:57225)
0808 22:54:05.356017 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.356030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:05.356104 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:05.356225 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:05.356226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:05.356227 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:05.356308 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.356310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.356923 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.356923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.358123 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:05.358125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.358843 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.358843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:05.358945 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.358951 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 22:54:05.359012 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:54:05.359060 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":138,"thread_start_us":122,"threads_started":1}
I20260808 22:54:05.359582 20020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7093ef09cb34ab5ae7133724f248ac6" candidate_uuid: "f9ad34d03207416ea4bec300e51ed1ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1e829f89fd4614b05eae6f2e6df580" is_pre_election: true
I20260808 22:54:05.359563 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7093ef09cb34ab5ae7133724f248ac6" candidate_uuid: "f9ad34d03207416ea4bec300e51ed1ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d839f8661f0a489a8b0d727aded9abb5" is_pre_election: true
I20260808 22:54:05.359736 20020 raft_consensus.cc:2454] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ad34d03207416ea4bec300e51ed1ee in term 0.
I20260808 22:54:05.359746 19596 raft_consensus.cc:2454] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ad34d03207416ea4bec300e51ed1ee in term 0.
I20260808 22:54:05.359969 19710 leader_election.cc:302] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b1e829f89fd4614b05eae6f2e6df580, f9ad34d03207416ea4bec300e51ed1ee; no voters:
I20260808 22:54:05.360100 20131 raft_consensus.cc:2781] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:05.360172 20131 raft_consensus.cc:482] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:05.360188 20131 raft_consensus.cc:3037] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:05.361038 20131 raft_consensus.cc:504] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.361173 20131 leader_election.cc:288] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [CANDIDATE]: Term 1 election: Requested vote from peers d839f8661f0a489a8b0d727aded9abb5 (127.22.30.1:43753), 3b1e829f89fd4614b05eae6f2e6df580 (127.22.30.3:34975)
I20260808 22:54:05.361370 20020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7093ef09cb34ab5ae7133724f248ac6" candidate_uuid: "f9ad34d03207416ea4bec300e51ed1ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1e829f89fd4614b05eae6f2e6df580"
I20260808 22:54:05.361359 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7093ef09cb34ab5ae7133724f248ac6" candidate_uuid: "f9ad34d03207416ea4bec300e51ed1ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d839f8661f0a489a8b0d727aded9abb5"
I20260808 22:54:05.361438 20020 raft_consensus.cc:3037] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:05.361440 19596 raft_consensus.cc:3037] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:05.362304 19596 raft_consensus.cc:2454] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ad34d03207416ea4bec300e51ed1ee in term 1.
I20260808 22:54:05.362304 20020 raft_consensus.cc:2454] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ad34d03207416ea4bec300e51ed1ee in term 1.
I20260808 22:54:05.362468 19703 leader_election.cc:302] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d839f8661f0a489a8b0d727aded9abb5, f9ad34d03207416ea4bec300e51ed1ee; no voters:
I20260808 22:54:05.362583 20131 raft_consensus.cc:2781] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:05.362718 20131 raft_consensus.cc:686] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 1 LEADER]: Becoming Leader. State: Replica: f9ad34d03207416ea4bec300e51ed1ee, State: Running, Role: LEADER
I20260808 22:54:05.362825 20131 consensus_queue.cc:237] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } }
I20260808 22:54:05.363618 19362 catalog_manager.cc:5495] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee reported cstate change: term changed from 0 to 1, leader changed from <none> to f9ad34d03207416ea4bec300e51ed1ee (127.22.30.2). New cstate: current_term: 1 leader_uuid: "f9ad34d03207416ea4bec300e51ed1ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ad34d03207416ea4bec300e51ed1ee" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 35421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260808 22:54:05.400341 20118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.398168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.22.30.2:35421)
0808 22:54:05.398219 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:05.398220 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:05.398247 (+ 27us) server_negotiation.cc:407] Connection header received
0808 22:54:05.398284 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:05.398284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:05.398300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:05.398346 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:05.398431 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.398932 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.399647 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:05.399894 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:05.399944 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:05.400068 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:05.400142 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:05.400203 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:05.400220 (+ 17us) server_negotiation.cc:299] Negotiation successful
0808 22:54:05.400227 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260808 22:54:05.402019 20020 raft_consensus.cc:1261] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 1 FOLLOWER]: Refusing update from remote peer f9ad34d03207416ea4bec300e51ed1ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:05.402045 19596 raft_consensus.cc:1261] T c7093ef09cb34ab5ae7133724f248ac6 P d839f8661f0a489a8b0d727aded9abb5 [term 1 FOLLOWER]: Refusing update from remote peer f9ad34d03207416ea4bec300e51ed1ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:05.402345 20137 consensus_queue.cc:1035] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1e829f89fd4614b05eae6f2e6df580" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:05.402459 20131 consensus_queue.cc:1035] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d839f8661f0a489a8b0d727aded9abb5" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:05.405306 20144 mvcc.cc:205] Tried to move back new op lower bound from 7316396627564298240 to 7316396627406700544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396627564298240}]
I20260808 22:54:05.405423 20145 mvcc.cc:205] Tried to move back new op lower bound from 7316396627564298240 to 7316396627406700544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396627564298240}]
I20260808 22:54:05.405500 20143 mvcc.cc:205] Tried to move back new op lower bound from 7316396627564298240 to 7316396627406700544. Current Snapshot: MvccSnapshot[applied={T|T < 7316396627564298240 or (T in {7316396627564298240})}]
I20260808 22:54:05.406849 19362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:05.406935 19362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:05.407660 19362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 731e8ddcf92b4d67a846beba0b303e07: Sending DeleteTablet for 3 replicas of tablet c7093ef09cb34ab5ae7133724f248ac6
I20260808 22:54:05.408008 19780 tablet_service.cc:1514] Processing DeleteTablet for tablet c7093ef09cb34ab5ae7133724f248ac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:05 UTC) from {username='slave'} at 127.0.0.1:41070
I20260808 22:54:05.408010 19562 tablet_service.cc:1514] Processing DeleteTablet for tablet c7093ef09cb34ab5ae7133724f248ac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:05 UTC) from {username='slave'} at 127.0.0.1:46024
I20260808 22:54:05.408016 19987 tablet_service.cc:1514] Processing DeleteTablet for tablet c7093ef09cb34ab5ae7133724f248ac6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:05 UTC) from {username='slave'} at 127.0.0.1:55972
I20260808 22:54:05.408227 20164 tablet_replica.cc:330] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: stopping tablet replica
I20260808 22:54:05.408321 20164 raft_consensus.cc:2229] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:05.408380 20167 tablet_replica.cc:330] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: stopping tablet replica
I20260808 22:54:05.408411 20164 raft_consensus.cc:2258] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:05.408439 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 19469
I20260808 22:54:05.408470 20167 raft_consensus.cc:2229] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:54:05.408550 20167 raft_consensus.cc:2258] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:05.408769 20164 ts_tablet_manager.cc:1905] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:05.410420 20164 ts_tablet_manager.cc:1918] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:05.410477 20164 log.cc:1196] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/c7093ef09cb34ab5ae7133724f248ac6
I20260808 22:54:05.410676 20164 ts_tablet_manager.cc:1939] T c7093ef09cb34ab5ae7133724f248ac6 P 3b1e829f89fd4614b05eae6f2e6df580: Deleting consensus metadata
I20260808 22:54:05.411072 19342 catalog_manager.cc:4841] TS 3b1e829f89fd4614b05eae6f2e6df580 (127.22.30.3:34975): tablet c7093ef09cb34ab5ae7133724f248ac6 (table test-table [id=7b525f66a1c5479090f1232e73feeaf1]) successfully deleted
I20260808 22:54:05.412438 20167 ts_tablet_manager.cc:1905] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:05.413962 20167 ts_tablet_manager.cc:1918] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:05.414026 20167 log.cc:1196] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/c7093ef09cb34ab5ae7133724f248ac6
I20260808 22:54:05.414283 20167 ts_tablet_manager.cc:1939] T c7093ef09cb34ab5ae7133724f248ac6 P f9ad34d03207416ea4bec300e51ed1ee: Deleting consensus metadata
I20260808 22:54:05.414623 19339 catalog_manager.cc:4841] TS f9ad34d03207416ea4bec300e51ed1ee (127.22.30.2:35421): tablet c7093ef09cb34ab5ae7133724f248ac6 (table test-table [id=7b525f66a1c5479090f1232e73feeaf1]) successfully deleted
W20260808 22:54:05.421120 19339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:43753 (error 108)
I20260808 22:54:05.421391 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 19677
I20260808 22:54:05.421551 20116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:05.421441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:43753 (local address 127.0.0.1:46034)
0808 22:54:05.421479 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:05.421517 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:43753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260808 22:54:05.421654 19339 catalog_manager.cc:4584] TS d839f8661f0a489a8b0d727aded9abb5 (127.22.30.1:43753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7093ef09cb34ab5ae7133724f248ac6: Network error: Client connection negotiation failed: client connection to 127.22.30.1:43753: connect: Connection refused (error 111)
I20260808 22:54:05.429301 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 19893
I20260808 22:54:05.439018 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 19317
2026-08-08T22:54:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (789 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:05 dist-test-slave-x4qh krb5kdc[20172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:05 dist-test-slave-x4qh krb5kdc[20172](info): set up 2 sockets
Aug 08 22:54:05 dist-test-slave-x4qh krb5kdc[20172](info): commencing operation
krb5kdc: starting...
W20260808 22:54:06.601557 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:06 dist-test-slave-x4qh krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:06.706359 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:36869
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:06.783429 20193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:06.783620 20193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:06.783654 20193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:06.784714 20193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:06.784772 20193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:06.784795 20193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:06.784828 20193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:06.784854 20193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:06.786281 20193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:36869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:06.786525 20193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:06.786734 20193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:06.791303 20201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:06.791313 20199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:06.791355 20203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:06.791579 20193 server_base.cc:1034] running on GCE node
I20260808 22:54:06.791827 20193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:06.792085 20193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:06.793242 20193 hybrid_clock.cc:648] HybridClock initialized: now 1786229646793225 us; error 41 us; skew 500 ppm
Aug 08 22:54:06 dist-test-slave-x4qh krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229646, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:06.795612 20193 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:06.796047 20193 webserver.cc:460] Webserver started at http://127.22.30.62:35871/ using document root <none> and password file <none>
I20260808 22:54:06.796273 20193 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:06.796339 20193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:06.796481 20193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:06.797427 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "200e187518704c32a8e318936aa5fe1c"
format_stamp: "Formatted at 2026-08-08 22:54:06 on dist-test-slave-x4qh"
server_key: "57570e3f3e77a932693139ad78e356f4"
server_key_iv: "60caf620e86c4f1e8fd86ecc3731be5a"
server_key_version: "encryptionkey@0"
I20260808 22:54:06.797783 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "200e187518704c32a8e318936aa5fe1c"
format_stamp: "Formatted at 2026-08-08 22:54:06 on dist-test-slave-x4qh"
server_key: "57570e3f3e77a932693139ad78e356f4"
server_key_iv: "60caf620e86c4f1e8fd86ecc3731be5a"
server_key_version: "encryptionkey@0"
I20260808 22:54:06.799247 20193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:06.800180 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:06.800345 20193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:06.800431 20193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "200e187518704c32a8e318936aa5fe1c"
format_stamp: "Formatted at 2026-08-08 22:54:06 on dist-test-slave-x4qh"
server_key: "57570e3f3e77a932693139ad78e356f4"
server_key_iv: "60caf620e86c4f1e8fd86ecc3731be5a"
server_key_version: "encryptionkey@0"
I20260808 22:54:06.800496 20193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:06.810232 20193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:06.811870 20193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:06.812011 20193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:06.815802 20193 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:36869
I20260808 22:54:06.815831 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:36869 every 8 connection(s)
I20260808 22:54:06.816160 20193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:06.816929 20301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:06.819013 20301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c: Bootstrap starting.
I20260808 22:54:06.819614 20301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:06.819895 20301 log.cc:824] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:06.820554 20301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c: No bootstrap required, opened a new log
I20260808 22:54:06.820909 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 20193
I20260808 22:54:06.821043 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:06.821262 22648 external_mini_cluster.cc:1428] Setting key 7d7d2415145d8318431b138752c97cde
I20260808 22:54:06.822644 20301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e187518704c32a8e318936aa5fe1c" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36869 } }
I20260808 22:54:06.822806 20301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:06.822844 20301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200e187518704c32a8e318936aa5fe1c, State: Initialized, Role: FOLLOWER
I20260808 22:54:06.823024 20301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e187518704c32a8e318936aa5fe1c" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36869 } }
I20260808 22:54:06.823117 20301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:06.823151 20301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:06.823202 20301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:06.824151 20301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e187518704c32a8e318936aa5fe1c" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36869 } }
I20260808 22:54:06.824261 20301 leader_election.cc:302] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200e187518704c32a8e318936aa5fe1c; no voters:
I20260808 22:54:06.824455 20301 leader_election.cc:288] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:06.824501 20308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:06.824668 20308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [term 1 LEADER]: Becoming Leader. State: Replica: 200e187518704c32a8e318936aa5fe1c, State: Running, Role: LEADER
I20260808 22:54:06.824700 20301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:06.824759 20308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e187518704c32a8e318936aa5fe1c" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36869 } }
I20260808 22:54:06.825120 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "200e187518704c32a8e318936aa5fe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e187518704c32a8e318936aa5fe1c" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36869 } } }
I20260808 22:54:06.825258 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:06.825433 20308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200e187518704c32a8e318936aa5fe1c. Latest consensus state: current_term: 1 leader_uuid: "200e187518704c32a8e318936aa5fe1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200e187518704c32a8e318936aa5fe1c" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 36869 } } }
I20260808 22:54:06.825541 20308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:06.826460 20323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:06.826946 20323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:06.828600 20323 catalog_manager.cc:1347] Generated new cluster ID: ea5b1a8840de4835859828e094024eb6
I20260808 22:54:06.828646 20323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:06 dist-test-slave-x4qh krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:06.832495 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:06.821713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.22.30.62:36869)
0808 22:54:06.821895 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:06.821901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:06.821924 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:54:06.822624 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:06.822631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:06.822673 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:06.822782 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:06.823612 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:06.824609 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:06.825676 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:06.826353 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:06.826453 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:06.829197 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:06.829201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:06.829201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:06.829211 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:06.830971 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:06.831319 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:06.831322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:06.831323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:06.831467 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:06.831665 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:06.831666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:06.831666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:06.831944 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:06.832116 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:06.832181 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 22:54:06.832313 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260808 22:54:06.869626 20323 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:06.870121 20323 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:06.874150 20323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c: Generated new TSK 0
I20260808 22:54:06.874343 20323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.30.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
Entry for principal kudu/127.22.30.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab.
I20260808 22:54:06.904211 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36869
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:06.981474 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:06.981648 20338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:06.981671 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:06.982667 20338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:06.982702 20338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:06.982724 20338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:54:06.984040 20338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.1.keytab
--principal=kudu/127.22.30.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:06.984192 20338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:06.984421 20338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:06.989148 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:06.989279 20338 server_base.cc:1034] running on GCE node
W20260808 22:54:06.989125 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:06.989135 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:06.989524 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:06.989787 20338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:06.990942 20338 hybrid_clock.cc:648] HybridClock initialized: now 1786229646990934 us; error 34 us; skew 500 ppm
Aug 08 22:54:06 dist-test-slave-x4qh krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229646, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:06.993193 20338 init.cc:377] Logged in from keytab as kudu/127.22.30.1@KRBTEST.COM (short username kudu)
I20260808 22:54:06.993546 20338 webserver.cc:460] Webserver started at http://127.22.30.1:36513/ using document root <none> and password file <none>
I20260808 22:54:06.993748 20338 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:06.993806 20338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:06.993934 20338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:06.994781 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "814c55aa5e98478d8ad90dd28cd5fec9"
format_stamp: "Formatted at 2026-08-08 22:54:06 on dist-test-slave-x4qh"
server_key: "33086b3b127d05faa2d3f719dd2c5677"
server_key_iv: "fac3c38fb0be0ef3bfa0ef6cd4b8be11"
server_key_version: "encryptionkey@0"
I20260808 22:54:06.995095 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "814c55aa5e98478d8ad90dd28cd5fec9"
format_stamp: "Formatted at 2026-08-08 22:54:06 on dist-test-slave-x4qh"
server_key: "33086b3b127d05faa2d3f719dd2c5677"
server_key_iv: "fac3c38fb0be0ef3bfa0ef6cd4b8be11"
server_key_version: "encryptionkey@0"
I20260808 22:54:06.996343 20338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:06.997151 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:06.997311 20338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:06.997399 20338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "814c55aa5e98478d8ad90dd28cd5fec9"
format_stamp: "Formatted at 2026-08-08 22:54:06 on dist-test-slave-x4qh"
server_key: "33086b3b127d05faa2d3f719dd2c5677"
server_key_iv: "fac3c38fb0be0ef3bfa0ef6cd4b8be11"
server_key_version: "encryptionkey@0"
I20260808 22:54:06.997474 20338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.004268 20338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.006235 20338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.006381 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.006589 20338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:07.006867 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:07.006918 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.006955 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:07.006983 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.012447 20338 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:45967
I20260808 22:54:07.012490 20548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:45967 every 8 connection(s)
I20260808 22:54:07.012745 20338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
Aug 08 22:54:07 dist-test-slave-x4qh krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229646, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.1@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:07.018573 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 20338
I20260808 22:54:07.018707 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:54:07.018873 22648 external_mini_cluster.cc:1428] Setting key 1922411138572fd088f9dd33f7067c5d
I20260808 22:54:07.021046 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.013330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:36951 (local address 127.22.30.62:36869)
0808 22:54:07.013371 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.013372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.013533 (+ 161us) server_negotiation.cc:407] Connection header received
0808 22:54:07.013977 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.013978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.013988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.014052 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.014802 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.015258 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.016183 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.016391 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.016442 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.018314 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.018315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.018315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:07.018321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.019826 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.020188 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.020189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.020190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.020245 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.020462 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.020463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.020463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.020564 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.020630 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.020868 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.020960 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:07.021180 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.013310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36869 (local address 127.22.30.1:36951)
0808 22:54:07.013439 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.013470 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.013897 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.014098 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.014100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.014567 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:07.014734 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.014738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.015333 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.015334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.016120 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.016122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.016460 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.016460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.016551 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.017006 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:07.017013 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:07.018229 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:07.019902 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:07.019909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:07.019911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:07.020106 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:07.020304 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:07.020305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:07.020305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:07.020396 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:07.020623 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:07.020630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:07.020746 (+ 116us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.020845 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.020979 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260808 22:54:07.021806 20550 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36869
I20260808 22:54:07.021883 20550 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:07.022058 20550 heartbeater.cc:511] Master 127.22.30.62:36869 requested a full tablet report, sending...
I20260808 22:54:07.022601 20245 ts_manager.cc:194] Registered new tserver with Master: 814c55aa5e98478d8ad90dd28cd5fec9 (127.22.30.1:45967)
I20260808 22:54:07.023439 20245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.1@KRBTEST.COM'} at 127.22.30.1:36951
WARNING: no policy specified for kudu/127.22.30.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
Entry for principal kudu/127.22.30.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab.
I20260808 22:54:07.042047 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36869
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:07.115453 20557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:07.115635 20557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:07.115671 20557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:07.116720 20557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:07.116786 20557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:07.116829 20557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:54:07.118284 20557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.2.keytab
--principal=kudu/127.22.30.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:07.118521 20557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:07.118767 20557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:07.123284 20570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.123289 20564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.123298 20566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.123580 20557 server_base.cc:1034] running on GCE node
I20260808 22:54:07.123751 20557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:07.123950 20557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:07.125084 20557 hybrid_clock.cc:648] HybridClock initialized: now 1786229647125064 us; error 32 us; skew 500 ppm
Aug 08 22:54:07 dist-test-slave-x4qh krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:07.127571 20557 init.cc:377] Logged in from keytab as kudu/127.22.30.2@KRBTEST.COM (short username kudu)
I20260808 22:54:07.127971 20557 webserver.cc:460] Webserver started at http://127.22.30.2:37775/ using document root <none> and password file <none>
I20260808 22:54:07.128181 20557 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:07.128221 20557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:07.128350 20557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:07.129220 20557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "ed7696cc31c94c37b0ec6974429eee3d"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "9305414bc725945a682b7aad7ac226c4"
server_key_iv: "1c8e8c8bdf4e5a22c5bf6971d4ad0684"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.129536 20557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "ed7696cc31c94c37b0ec6974429eee3d"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "9305414bc725945a682b7aad7ac226c4"
server_key_iv: "1c8e8c8bdf4e5a22c5bf6971d4ad0684"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.130668 20557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:07.131285 20583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.131477 20557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.131562 20557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "ed7696cc31c94c37b0ec6974429eee3d"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "9305414bc725945a682b7aad7ac226c4"
server_key_iv: "1c8e8c8bdf4e5a22c5bf6971d4ad0684"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.131639 20557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.144680 20557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.146697 20557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.146870 20557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.147065 20557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:07.147332 20557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:07.147399 20557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.147437 20557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:07.147466 20557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.153033 20557 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:40621
I20260808 22:54:07.153079 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:40621 every 8 connection(s)
I20260808 22:54:07.153327 20557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:54:07.156279 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 20557
I20260808 22:54:07.156410 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:54:07.156584 22648 external_mini_cluster.cc:1428] Setting key b92f6b61ed0fbe704201508750e80cee
Aug 08 22:54:07 dist-test-slave-x4qh krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.2@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:07.162228 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.153838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36869 (local address 127.22.30.2:33041)
0808 22:54:07.153988 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.154017 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.154385 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.154587 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.154590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.155069 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:07.155221 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.155224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.155832 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.155833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.156727 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.156730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.157183 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.157184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.157353 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.157744 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:07.157753 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:07.159351 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:07.160996 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:07.161002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:07.161003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:07.161193 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:07.161407 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:07.161408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:07.161408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:07.161526 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:07.161802 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:07.161808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:07.161930 (+ 122us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.162003 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.162101 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260808 22:54:07.162233 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.153871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:33041 (local address 127.22.30.62:36869)
0808 22:54:07.153933 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.153936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.154084 (+ 148us) server_negotiation.cc:407] Connection header received
0808 22:54:07.154477 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.154479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.154487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.154520 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.155288 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.155755 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.156811 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.157074 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.157141 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.159455 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.159456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.159457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:07.159463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.160912 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.161259 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.161260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.161261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.161328 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.161587 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.161588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.161588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.161722 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.161795 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.162051 (+ 256us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.162135 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":35}
I20260808 22:54:07.162636 20775 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36869
I20260808 22:54:07.162734 20775 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:07.162906 20775 heartbeater.cc:511] Master 127.22.30.62:36869 requested a full tablet report, sending...
I20260808 22:54:07.163244 20245 ts_manager.cc:194] Registered new tserver with Master: ed7696cc31c94c37b0ec6974429eee3d (127.22.30.2:40621)
I20260808 22:54:07.163614 20245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.2@KRBTEST.COM'} at 127.22.30.2:33041
WARNING: no policy specified for kudu/127.22.30.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
Entry for principal kudu/127.22.30.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab.
I20260808 22:54:07.181561 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36869
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:07.257436 20786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:07.257661 20786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:07.257699 20786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:07.258742 20786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:07.258797 20786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:07.258839 20786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:54:07.260252 20786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.3.keytab
--principal=kudu/127.22.30.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:36869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:07.260483 20786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:07.260710 20786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:07.265254 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.265275 20800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.265290 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.265570 20786 server_base.cc:1034] running on GCE node
I20260808 22:54:07.265740 20786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:07.265954 20786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:07.267105 20786 hybrid_clock.cc:648] HybridClock initialized: now 1786229647267091 us; error 26 us; skew 500 ppm
Aug 08 22:54:07 dist-test-slave-x4qh krb5kdc[20172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:07.269495 20786 init.cc:377] Logged in from keytab as kudu/127.22.30.3@KRBTEST.COM (short username kudu)
I20260808 22:54:07.269881 20786 webserver.cc:460] Webserver started at http://127.22.30.3:35421/ using document root <none> and password file <none>
I20260808 22:54:07.270083 20786 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:07.270141 20786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:07.270268 20786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:07.271203 20786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "758fac19518f4cd2b758eae62b6d4346"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "07ca3621808c36cbc86aa4413d621184"
server_key_iv: "39a0b2513948e5103613fb8a842c517c"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.271562 20786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "758fac19518f4cd2b758eae62b6d4346"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "07ca3621808c36cbc86aa4413d621184"
server_key_iv: "39a0b2513948e5103613fb8a842c517c"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.272833 20786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:54:07.273491 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.273638 20786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:07.273722 20786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "758fac19518f4cd2b758eae62b6d4346"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "07ca3621808c36cbc86aa4413d621184"
server_key_iv: "39a0b2513948e5103613fb8a842c517c"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.273801 20786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.284845 20786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.286972 20786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.287119 20786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.287335 20786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:07.287634 20786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:07.287688 20786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.287717 20786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:07.287741 20786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.293097 20786 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:33941
I20260808 22:54:07.293130 20988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:33941 every 8 connection(s)
I20260808 22:54:07.293416 20786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:54:07.295840 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 20786
I20260808 22:54:07.295969 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:54:07.296157 22648 external_mini_cluster.cc:1428] Setting key 2de01c0baaa61ce1e2408e6b17483bae
Aug 08 22:54:07 dist-test-slave-x4qh krb5kdc[20172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229647, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.3@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:07.302443 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.294064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:36617 (local address 127.22.30.62:36869)
0808 22:54:07.294132 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.294134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.294345 (+ 211us) server_negotiation.cc:407] Connection header received
0808 22:54:07.294870 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.294871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.294883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.294926 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.295922 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.296632 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.297808 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.298006 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.298059 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.299966 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.299967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.299968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:07.299974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.301337 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.301662 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.301662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.301663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.301711 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.301873 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.301873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.301874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.301965 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.302014 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.302321 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.302382 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 22:54:07.302551 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.294021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:36869 (local address 127.22.30.3:36617)
0808 22:54:07.294247 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.294271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.294765 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.294992 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.294996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.295551 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:07.295794 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.295801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.296714 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.296714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.297722 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.297725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.298075 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.298075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.298165 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.298547 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:07.298554 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:07.299883 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:07.301402 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:07.301407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:07.301409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:07.301601 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:07.301750 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:07.301751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:07.301751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:07.301827 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:07.302058 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:07.302063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:07.302164 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.302301 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.302413 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260808 22:54:07.303053 20992 heartbeater.cc:347] Connected to a master server at 127.22.30.62:36869
I20260808 22:54:07.303141 20992 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:07.303282 20992 heartbeater.cc:511] Master 127.22.30.62:36869 requested a full tablet report, sending...
I20260808 22:54:07.303653 20245 ts_manager.cc:194] Registered new tserver with Master: 758fac19518f4cd2b758eae62b6d4346 (127.22.30.3:33941)
I20260808 22:54:07.303977 20245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.30.3@KRBTEST.COM'} at 127.22.30.3:36617
I20260808 22:54:07.307809 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:54:07.314118 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.308895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.22.30.62:36869)
0808 22:54:07.308943 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.308945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.309051 (+ 106us) server_negotiation.cc:407] Connection header received
0808 22:54:07.309118 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.309119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.309128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.309188 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.309718 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.310123 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.310799 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.310963 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.311023 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.311990 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.311992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.311992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:07.311998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.313070 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.313383 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.313384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.313384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.313435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:07.313612 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:07.313613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:07.313613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:07.313757 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.313859 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.313960 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.314031 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:54:07.315531 20245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:54:07.317057 20245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:54:07.322664 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.318917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:33941 (local address 127.0.0.1:44148)
0808 22:54:07.319286 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.319322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.319441 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.319688 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.319688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.319690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:07.319720 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.319721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.320851 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.320853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.321699 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.321702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.322402 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.322403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.322473 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.322486 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.322554 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.322599 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":88,"thread_start_us":60,"threads_started":1}
I20260808 22:54:07.322681 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.319147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.22.30.3:33941)
0808 22:54:07.319308 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.319312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.319421 (+ 109us) server_negotiation.cc:407] Connection header received
0808 22:54:07.319516 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.319518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.319532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.319611 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:07.319795 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.320747 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.321785 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.322327 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.322394 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.322440 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.322577 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.322614 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260808 22:54:07.323153 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.318918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40621 (local address 127.0.0.1:48076)
0808 22:54:07.319352 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.319361 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.319455 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.319648 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.319652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.319654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:07.319719 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.319721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.320928 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.320929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.322000 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.322002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.322949 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.322950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.323009 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.323012 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.323057 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.323093 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":20,"thread_start_us":50,"threads_started":1}
I20260808 22:54:07.323434 20888 tablet_service.cc:1467] Processing CreateTablet for tablet b0260debd2a54bac87216510b2fa4f70 (DEFAULT_TABLE table=test-table [id=2bf50f7d4ba74d558551ac4122acffc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:07.323575 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.318924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:45967 (local address 127.0.0.1:36726)
0808 22:54:07.319544 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.319552 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.319587 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.319807 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.319808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.319811 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:07.319856 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.319858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.320798 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.320799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.322228 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.322230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.323297 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.323298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.323356 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.323359 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.323429 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.323486 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":74,"thread_start_us":80,"threads_started":1}
I20260808 22:54:07.323758 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.319149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.22.30.1:45967)
0808 22:54:07.319583 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.319587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.319597 (+ 10us) server_negotiation.cc:407] Connection header received
0808 22:54:07.319650 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.319652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.319665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.319728 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:07.319935 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.320708 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.322306 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.323155 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.323236 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.323298 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.323596 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.323639 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":106,"threads_started":1}
I20260808 22:54:07.323783 20888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0260debd2a54bac87216510b2fa4f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:07.324394 20427 tablet_service.cc:1467] Processing CreateTablet for tablet b0260debd2a54bac87216510b2fa4f70 (DEFAULT_TABLE table=test-table [id=2bf50f7d4ba74d558551ac4122acffc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:07.324734 20427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0260debd2a54bac87216510b2fa4f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:07.327066 21012 tablet_bootstrap.cc:492] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Bootstrap starting.
I20260808 22:54:07.327345 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.319080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.22.30.2:40621)
0808 22:54:07.319272 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.319277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.319450 (+ 173us) server_negotiation.cc:407] Connection header received
0808 22:54:07.319517 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.319519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.319532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.319796 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:07.319874 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.320841 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.322073 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.322860 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.322936 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.322988 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.327228 (+ 4240us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.327272 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260808 22:54:07.327771 21012 tablet_bootstrap.cc:654] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:07.327905 20663 tablet_service.cc:1467] Processing CreateTablet for tablet b0260debd2a54bac87216510b2fa4f70 (DEFAULT_TABLE table=test-table [id=2bf50f7d4ba74d558551ac4122acffc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:07.328096 21013 tablet_bootstrap.cc:492] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: Bootstrap starting.
I20260808 22:54:07.328244 20663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0260debd2a54bac87216510b2fa4f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:07.328434 21012 log.cc:824] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:07.328831 21013 tablet_bootstrap.cc:654] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:07.329209 21012 tablet_bootstrap.cc:492] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: No bootstrap required, opened a new log
I20260808 22:54:07.329293 21012 ts_tablet_manager.cc:1397] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:07.329507 21013 log.cc:824] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:07.330437 21013 tablet_bootstrap.cc:492] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: No bootstrap required, opened a new log
I20260808 22:54:07.330580 21013 ts_tablet_manager.cc:1397] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:54:07.330819 21018 tablet_bootstrap.cc:492] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Bootstrap starting.
I20260808 22:54:07.331233 21012 raft_consensus.cc:348] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.331359 21018 tablet_bootstrap.cc:654] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:07.331447 21012 raft_consensus.cc:374] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:07.331496 21012 raft_consensus.cc:729] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758fac19518f4cd2b758eae62b6d4346, State: Initialized, Role: FOLLOWER
I20260808 22:54:07.331638 21012 consensus_queue.cc:260] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.331918 21012 ts_tablet_manager.cc:1428] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:07.331988 21018 log.cc:824] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:07.332029 20992 heartbeater.cc:503] Master 127.22.30.62:36869 was elected leader, sending a full tablet report...
I20260808 22:54:07.332640 21013 raft_consensus.cc:348] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.332828 21013 raft_consensus.cc:374] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:07.332863 21013 raft_consensus.cc:729] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814c55aa5e98478d8ad90dd28cd5fec9, State: Initialized, Role: FOLLOWER
I20260808 22:54:07.332980 21013 consensus_queue.cc:260] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.333227 21013 ts_tablet_manager.cc:1428] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 22:54:07.333268 20550 heartbeater.cc:503] Master 127.22.30.62:36869 was elected leader, sending a full tablet report...
I20260808 22:54:07.333793 21018 tablet_bootstrap.cc:492] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: No bootstrap required, opened a new log
I20260808 22:54:07.333851 21018 ts_tablet_manager.cc:1397] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 22:54:07.335361 21018 raft_consensus.cc:348] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.335529 21018 raft_consensus.cc:374] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:07.335551 21018 raft_consensus.cc:729] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7696cc31c94c37b0ec6974429eee3d, State: Initialized, Role: FOLLOWER
I20260808 22:54:07.335638 21018 consensus_queue.cc:260] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.335840 21018 ts_tablet_manager.cc:1428] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 22:54:07.335903 20775 heartbeater.cc:503] Master 127.22.30.62:36869 was elected leader, sending a full tablet report...
I20260808 22:54:07.389206 21027 raft_consensus.cc:482] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:07.389285 21027 raft_consensus.cc:504] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.389622 21027 leader_election.cc:288] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 758fac19518f4cd2b758eae62b6d4346 (127.22.30.3:33941), ed7696cc31c94c37b0ec6974429eee3d (127.22.30.2:40621)
I20260808 22:54:07.393348 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.389862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:43759 (local address 127.22.30.3:33941)
0808 22:54:07.389930 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.389932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.389941 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:07.389978 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.389979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.389988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.390040 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:07.390223 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.391018 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.392537 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.393044 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.393111 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.393120 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.393258 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.393292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 22:54:07.393348 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.389878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:35909 (local address 127.22.30.2:40621)
0808 22:54:07.389922 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.389924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.390097 (+ 173us) server_negotiation.cc:407] Connection header received
0808 22:54:07.390138 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.390138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.390146 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.390208 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:07.390335 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.391026 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.392537 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.393044 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.393111 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.393120 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.393258 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.393297 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:54:07.393390 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.389799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:33941 (local address 127.22.30.1:43759)
0808 22:54:07.389861 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.389872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.389926 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.390068 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.390069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.390071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:07.390140 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.390142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.391126 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.391127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.392462 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.392465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.393119 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.393120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.393188 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.393192 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.393247 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.393290 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":81}
I20260808 22:54:07.393402 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.389799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:40621 (local address 127.22.30.1:35909)
0808 22:54:07.390013 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.390035 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.390087 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.390214 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.390215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.390217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:07.390277 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.390278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.391126 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.391127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.392462 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.392465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.393119 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.393120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.393188 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.393192 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.393245 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.393289 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":115,"thread_start_us":104,"threads_started":1}
I20260808 22:54:07.393671 20915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0260debd2a54bac87216510b2fa4f70" candidate_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758fac19518f4cd2b758eae62b6d4346" is_pre_election: true
I20260808 22:54:07.393703 20702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0260debd2a54bac87216510b2fa4f70" candidate_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7696cc31c94c37b0ec6974429eee3d" is_pre_election: true
I20260808 22:54:07.393802 20915 raft_consensus.cc:2454] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c55aa5e98478d8ad90dd28cd5fec9 in term 0.
I20260808 22:54:07.393801 20702 raft_consensus.cc:2454] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 814c55aa5e98478d8ad90dd28cd5fec9 in term 0.
I20260808 22:54:07.393980 20359 leader_election.cc:302] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758fac19518f4cd2b758eae62b6d4346, 814c55aa5e98478d8ad90dd28cd5fec9; no voters:
I20260808 22:54:07.394089 21027 raft_consensus.cc:2781] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:07.394151 21027 raft_consensus.cc:482] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:07.394167 21027 raft_consensus.cc:3037] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:07.394783 21027 raft_consensus.cc:504] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.394944 21027 leader_election.cc:288] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [CANDIDATE]: Term 1 election: Requested vote from peers 758fac19518f4cd2b758eae62b6d4346 (127.22.30.3:33941), ed7696cc31c94c37b0ec6974429eee3d (127.22.30.2:40621)
I20260808 22:54:07.395093 20702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0260debd2a54bac87216510b2fa4f70" candidate_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7696cc31c94c37b0ec6974429eee3d"
I20260808 22:54:07.395112 20915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0260debd2a54bac87216510b2fa4f70" candidate_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758fac19518f4cd2b758eae62b6d4346"
I20260808 22:54:07.395171 20702 raft_consensus.cc:3037] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:07.395202 20915 raft_consensus.cc:3037] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:07.395898 20702 raft_consensus.cc:2454] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814c55aa5e98478d8ad90dd28cd5fec9 in term 1.
I20260808 22:54:07.395905 20915 raft_consensus.cc:2454] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 814c55aa5e98478d8ad90dd28cd5fec9 in term 1.
I20260808 22:54:07.396085 20361 leader_election.cc:302] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 814c55aa5e98478d8ad90dd28cd5fec9, ed7696cc31c94c37b0ec6974429eee3d; no voters:
I20260808 22:54:07.396212 21027 raft_consensus.cc:2781] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:07.396391 21027 raft_consensus.cc:686] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 1 LEADER]: Becoming Leader. State: Replica: 814c55aa5e98478d8ad90dd28cd5fec9, State: Running, Role: LEADER
I20260808 22:54:07.396478 21027 consensus_queue.cc:237] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } }
I20260808 22:54:07.397190 20245 catalog_manager.cc:5495] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 814c55aa5e98478d8ad90dd28cd5fec9 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "814c55aa5e98478d8ad90dd28cd5fec9" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 45967 } health_report { overall_health: HEALTHY } } }
W20260808 22:54:07.403760 20778 tablet.cc:2369] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:54:07.412848 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.410876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.22.30.1:45967)
0808 22:54:07.410947 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.410963 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.410992 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:54:07.411017 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.411017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.411030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.411078 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:07.411202 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.411660 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.412317 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.412472 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.412518 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.412629 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:07.412724 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:07.412778 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.412790 (+ 12us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.412806 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 22:54:07.414305 20915 raft_consensus.cc:1261] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 1 FOLLOWER]: Refusing update from remote peer 814c55aa5e98478d8ad90dd28cd5fec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:07.414323 20702 raft_consensus.cc:1261] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 1 FOLLOWER]: Refusing update from remote peer 814c55aa5e98478d8ad90dd28cd5fec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:07.414615 21030 consensus_queue.cc:1035] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "758fac19518f4cd2b758eae62b6d4346" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:07.414727 21027 consensus_queue.cc:1035] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7696cc31c94c37b0ec6974429eee3d" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:07.416910 21033 mvcc.cc:205] Tried to move back new op lower bound from 7316396635806679040 to 7316396635736453120. Current Snapshot: MvccSnapshot[applied={T|T < 7316396635806679040}]
I20260808 22:54:07.417032 21034 mvcc.cc:205] Tried to move back new op lower bound from 7316396635806679040 to 7316396635736453120. Current Snapshot: MvccSnapshot[applied={T|T < 7316396635806679040}]
I20260808 22:54:07.418840 20242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:07.418908 20242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:07.419520 20242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 200e187518704c32a8e318936aa5fe1c: Sending DeleteTablet for 3 replicas of tablet b0260debd2a54bac87216510b2fa4f70
I20260808 22:54:07.419790 20888 tablet_service.cc:1514] Processing DeleteTablet for tablet b0260debd2a54bac87216510b2fa4f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:07 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:44148
I20260808 22:54:07.419857 20427 tablet_service.cc:1514] Processing DeleteTablet for tablet b0260debd2a54bac87216510b2fa4f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:07 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:36726
I20260808 22:54:07.420006 20663 tablet_service.cc:1514] Processing DeleteTablet for tablet b0260debd2a54bac87216510b2fa4f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:07 UTC) from {username='kudu', principal='kudu/127.22.30.62@KRBTEST.COM'} at 127.0.0.1:48076
I20260808 22:54:07.420063 21047 tablet_replica.cc:330] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9: stopping tablet replica
I20260808 22:54:07.420148 21047 raft_consensus.cc:2229] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 22:54:07.420228 21047 raft_consensus.cc:2258] T b0260debd2a54bac87216510b2fa4f70 P 814c55aa5e98478d8ad90dd28cd5fec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:07.420372 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 20338
I20260808 22:54:07.430781 21046 tablet_replica.cc:330] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: stopping tablet replica
I20260808 22:54:07.430902 21046 raft_consensus.cc:2229] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:07.431017 21046 raft_consensus.cc:2258] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:07.431018 21048 tablet_replica.cc:330] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: stopping tablet replica
I20260808 22:54:07.431137 21048 raft_consensus.cc:2229] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:07.431246 21048 raft_consensus.cc:2258] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:07.432214 21046 ts_tablet_manager.cc:1905] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:07.433580 21048 ts_tablet_manager.cc:1905] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:07.433769 21046 ts_tablet_manager.cc:1918] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:07.433869 21046 log.cc:1196] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/b0260debd2a54bac87216510b2fa4f70
I20260808 22:54:07.434167 21046 ts_tablet_manager.cc:1939] T b0260debd2a54bac87216510b2fa4f70 P 758fac19518f4cd2b758eae62b6d4346: Deleting consensus metadata
I20260808 22:54:07.434572 20219 catalog_manager.cc:4841] TS 758fac19518f4cd2b758eae62b6d4346 (127.22.30.3:33941): tablet b0260debd2a54bac87216510b2fa4f70 (table test-table [id=2bf50f7d4ba74d558551ac4122acffc0]) successfully deleted
I20260808 22:54:07.435339 21048 ts_tablet_manager.cc:1918] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:07.435447 21048 log.cc:1196] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/b0260debd2a54bac87216510b2fa4f70
I20260808 22:54:07.435684 21048 ts_tablet_manager.cc:1939] T b0260debd2a54bac87216510b2fa4f70 P ed7696cc31c94c37b0ec6974429eee3d: Deleting consensus metadata
I20260808 22:54:07.436059 20221 catalog_manager.cc:4841] TS ed7696cc31c94c37b0ec6974429eee3d (127.22.30.2:40621): tablet b0260debd2a54bac87216510b2fa4f70 (table test-table [id=2bf50f7d4ba74d558551ac4122acffc0]) successfully deleted
W20260808 22:54:07.438663 20222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:45967 (error 108)
I20260808 22:54:07.438908 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 20557
I20260808 22:54:07.439062 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.438951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:45967 (local address 127.0.0.1:36736)
0808 22:54:07.438999 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.439030 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:45967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 22:54:07.439153 20222 catalog_manager.cc:4584] TS 814c55aa5e98478d8ad90dd28cd5fec9 (127.22.30.1:45967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0260debd2a54bac87216510b2fa4f70: Network error: Client connection negotiation failed: client connection to 127.22.30.1:45967: connect: Connection refused (error 111)
I20260808 22:54:07.446257 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 20786
I20260808 22:54:07.455596 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 20193
2026-08-08T22:54:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2017 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T22:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:07Z Disabled control of system clock
I20260808 22:54:07.486778 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43811
--webserver_interface=127.22.30.62
--webserver_port=0
--builtin_ntp_servers=127.22.30.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:43811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:07.562947 21058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:07.563124 21058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:07.563143 21058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:07.564148 21058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:07.564203 21058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:07.564217 21058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:07.564229 21058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:07.564241 21058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:07.565507 21058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:43811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:07.565682 21058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:07.565896 21058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:07.570309 21069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.570519 21058 server_base.cc:1034] running on GCE node
W20260808 22:54:07.570384 21066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.570309 21067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.570886 21058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:07.571143 21058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:07.572291 21058 hybrid_clock.cc:648] HybridClock initialized: now 1786229647572283 us; error 34 us; skew 500 ppm
I20260808 22:54:07.573354 21058 webserver.cc:460] Webserver started at http://127.22.30.62:42021/ using document root <none> and password file <none>
I20260808 22:54:07.573562 21058 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:07.573621 21058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:07.573742 21058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:07.574697 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "8eceaf1f70fd48d5852cecaa30dce1ae"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "eb432feba16ac559e24219ecbdc2b7b0"
server_key_iv: "be2ec4bc56c2b6d419b6bffd84d6a34f"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.575042 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "8eceaf1f70fd48d5852cecaa30dce1ae"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "eb432feba16ac559e24219ecbdc2b7b0"
server_key_iv: "be2ec4bc56c2b6d419b6bffd84d6a34f"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.576385 21058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:07.577064 21078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.577219 21058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:07.577322 21058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "8eceaf1f70fd48d5852cecaa30dce1ae"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "eb432feba16ac559e24219ecbdc2b7b0"
server_key_iv: "be2ec4bc56c2b6d419b6bffd84d6a34f"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.577392 21058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.583505 21058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.583734 21058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.583868 21058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.587451 21058 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:43811
I20260808 22:54:07.587491 21151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:43811 every 8 connection(s)
I20260808 22:54:07.587765 21058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:07.588580 21152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:07.590338 21152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae: Bootstrap starting.
I20260808 22:54:07.590698 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 21058
I20260808 22:54:07.590804 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:07.590806 21152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:07.590972 22648 external_mini_cluster.cc:1428] Setting key c16905c18b40ef73c86833c697e89d9a
I20260808 22:54:07.591137 21152 log.cc:824] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:07.591869 21152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae: No bootstrap required, opened a new log
I20260808 22:54:07.594066 21152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43811 } }
I20260808 22:54:07.594177 21152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:07.594197 21152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eceaf1f70fd48d5852cecaa30dce1ae, State: Initialized, Role: FOLLOWER
I20260808 22:54:07.594305 21152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43811 } }
I20260808 22:54:07.594372 21152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:07.594406 21152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:07.594450 21152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:07.595070 21152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43811 } }
I20260808 22:54:07.595180 21152 leader_election.cc:302] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eceaf1f70fd48d5852cecaa30dce1ae; no voters:
I20260808 22:54:07.595335 21152 leader_election.cc:288] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:07.595500 21159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:07.595666 21159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [term 1 LEADER]: Becoming Leader. State: Replica: 8eceaf1f70fd48d5852cecaa30dce1ae, State: Running, Role: LEADER
I20260808 22:54:07.595752 21152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:07.595767 21159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43811 } }
I20260808 22:54:07.596371 21161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43811 } } }
I20260808 22:54:07.596455 21161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:07.596604 21164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eceaf1f70fd48d5852cecaa30dce1ae. Latest consensus state: current_term: 1 leader_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eceaf1f70fd48d5852cecaa30dce1ae" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 43811 } } }
I20260808 22:54:07.596679 21164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:07.596652 21175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:07.597280 21175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:07.597595 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.591449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.22.30.62:43811)
0808 22:54:07.591612 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.591619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.591648 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:54:07.592142 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.592149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.592214 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.592359 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.592506 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.593782 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.594624 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.595213 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.595342 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.595989 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.595992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.595992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:07.596182 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.596224 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:07.596237 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.597225 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.597256 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.597341 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
I20260808 22:54:07.598737 21175 catalog_manager.cc:1347] Generated new cluster ID: 38e54e550b0d4313abdd25ef94fcac2a
I20260808 22:54:07.598800 21175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:07.619757 21175 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:07.620225 21175 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:07.629235 21175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae: Generated new TSK 0
I20260808 22:54:07.629385 21175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:07.639252 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.1:0
--local_ip_for_outbound_sockets=127.22.30.1
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43811
--builtin_ntp_servers=127.22.30.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:07.715746 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:07.715938 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:07.715972 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:07.717041 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:07.717100 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:07.717144 21186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.1
I20260808 22:54:07.718385 21186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.30.1
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.1
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:07.718618 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:07.718883 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:07.723515 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.723529 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.723546 21186 server_base.cc:1034] running on GCE node
W20260808 22:54:07.723534 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.723892 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:07.724104 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:07.725255 21186 hybrid_clock.cc:648] HybridClock initialized: now 1786229647725247 us; error 36 us; skew 500 ppm
I20260808 22:54:07.726366 21186 webserver.cc:460] Webserver started at http://127.22.30.1:34159/ using document root <none> and password file <none>
I20260808 22:54:07.726651 21186 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:07.726720 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:07.726855 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:07.727933 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data/instance:
uuid: "915b741547ca4766a3df84e82d80b005"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "e25ab57351c36d4b3fbcad982f6c8b58"
server_key_iv: "2678de73eeb510dd33d1a3f7549ec036"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.728288 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance:
uuid: "915b741547ca4766a3df84e82d80b005"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "e25ab57351c36d4b3fbcad982f6c8b58"
server_key_iv: "2678de73eeb510dd33d1a3f7549ec036"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.729423 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:07.730064 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.730223 21186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:07.730312 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal
uuid: "915b741547ca4766a3df84e82d80b005"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "e25ab57351c36d4b3fbcad982f6c8b58"
server_key_iv: "2678de73eeb510dd33d1a3f7549ec036"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.730384 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.738899 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.739144 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.739269 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.739496 21186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:07.739778 21186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:07.739826 21186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.739862 21186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:07.739888 21186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.745411 21186 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.1:37261
I20260808 22:54:07.745460 21376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.1:37261 every 8 connection(s)
I20260808 22:54:07.745769 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/data/info.pb
I20260808 22:54:07.750391 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.746500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:59341 (local address 127.22.30.62:43811)
0808 22:54:07.746566 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.746569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.746788 (+ 219us) server_negotiation.cc:407] Connection header received
0808 22:54:07.747262 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.747263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.747272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.747316 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.747960 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.748453 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.749357 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.749520 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.749564 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.750213 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.750214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.750215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:07.750218 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.750230 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:07.750233 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.750282 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.750320 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.750339 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 22:54:07.750631 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.746459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43811 (local address 127.22.30.1:59341)
0808 22:54:07.746679 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.746711 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.747170 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.747424 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.747429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.747438 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:07.747891 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.747897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.748532 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.748532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.749292 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.749293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.749586 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.749587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.749687 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.750090 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:07.750091 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:07.750125 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:07.750144 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:07.750279 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:07.750281 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:07.750283 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.750322 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.750389 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260808 22:54:07.751178 21377 heartbeater.cc:347] Connected to a master server at 127.22.30.62:43811
I20260808 22:54:07.751276 21377 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:07.751499 21377 heartbeater.cc:511] Master 127.22.30.62:43811 requested a full tablet report, sending...
I20260808 22:54:07.751983 21102 ts_manager.cc:194] Registered new tserver with Master: 915b741547ca4766a3df84e82d80b005 (127.22.30.1:37261)
I20260808 22:54:07.752612 21102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.1:59341
I20260808 22:54:07.754406 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 21186
I20260808 22:54:07.754495 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-0/wal/instance
I20260808 22:54:07.754650 22648 external_mini_cluster.cc:1428] Setting key c8709f597be94761159687b20546a172
I20260808 22:54:07.755342 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.2:0
--local_ip_for_outbound_sockets=127.22.30.2
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43811
--builtin_ntp_servers=127.22.30.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:07.831239 21382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:07.831411 21382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:07.831430 21382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:07.832379 21382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:07.832430 21382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:07.832477 21382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.2
I20260808 22:54:07.833797 21382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.30.2
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.2
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:07.834007 21382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:07.834312 21382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:07.838907 21400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.838999 21395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.839020 21382 server_base.cc:1034] running on GCE node
W20260808 22:54:07.838917 21394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.839279 21382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:07.839535 21382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:07.840688 21382 hybrid_clock.cc:648] HybridClock initialized: now 1786229647840680 us; error 35 us; skew 500 ppm
I20260808 22:54:07.841806 21382 webserver.cc:460] Webserver started at http://127.22.30.2:42333/ using document root <none> and password file <none>
I20260808 22:54:07.842046 21382 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:07.842110 21382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:07.842237 21382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:07.843138 21382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data/instance:
uuid: "cb37bb26c08745028abd8760b2d767ce"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "c1d7f8d33f0f2c1e9aa95fa745bcb423"
server_key_iv: "69b99ae41b30bb66831cce327402c5a8"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.843492 21382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance:
uuid: "cb37bb26c08745028abd8760b2d767ce"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "c1d7f8d33f0f2c1e9aa95fa745bcb423"
server_key_iv: "69b99ae41b30bb66831cce327402c5a8"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.844651 21382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:54:07.845323 21406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.845472 21382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.845558 21382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal
uuid: "cb37bb26c08745028abd8760b2d767ce"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "c1d7f8d33f0f2c1e9aa95fa745bcb423"
server_key_iv: "69b99ae41b30bb66831cce327402c5a8"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.845631 21382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.854444 21382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.854694 21382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.854835 21382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.855027 21382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:07.855314 21382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:07.855365 21382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.855427 21382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:07.855455 21382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.861016 21382 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.2:37873
I20260808 22:54:07.861063 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.2:37873 every 8 connection(s)
I20260808 22:54:07.861315 21382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/data/info.pb
I20260808 22:54:07.865934 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.861929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.2:54785 (local address 127.22.30.62:43811)
0808 22:54:07.861998 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.862000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.862181 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:54:07.862596 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.862597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.862608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.862654 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.863225 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.863744 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.864667 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.864877 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.864947 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.865687 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.865688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.865689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:07.865693 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.865708 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:07.865711 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.865757 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.865836 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.865862 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 22:54:07.866155 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.861901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43811 (local address 127.22.30.2:54785)
0808 22:54:07.862075 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.862102 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.862503 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.862715 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.862718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.862725 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:07.863142 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.863146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.863834 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.863835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.864557 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.864559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.864960 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.864961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.865080 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.865548 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:07.865549 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:07.865572 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:07.865592 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:07.865757 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:07.865761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:07.865764 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.865837 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.865927 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260808 22:54:07.866626 21583 heartbeater.cc:347] Connected to a master server at 127.22.30.62:43811
I20260808 22:54:07.866698 21583 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:07.866820 21583 heartbeater.cc:511] Master 127.22.30.62:43811 requested a full tablet report, sending...
I20260808 22:54:07.867125 21102 ts_manager.cc:194] Registered new tserver with Master: cb37bb26c08745028abd8760b2d767ce (127.22.30.2:37873)
I20260808 22:54:07.867545 21102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.2:54785
I20260808 22:54:07.870303 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 21382
I20260808 22:54:07.870429 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal/instance
I20260808 22:54:07.870628 22648 external_mini_cluster.cc:1428] Setting key ebfdd2f915250634b083758d6f969e09
I20260808 22:54:07.871448 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.30.3:0
--local_ip_for_outbound_sockets=127.22.30.3
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43811
--builtin_ntp_servers=127.22.30.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 22:54:07.950250 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:07.950446 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:07.950469 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:07.951612 21586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:07.951655 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:07.951683 21586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.30.3
I20260808 22:54:07.953133 21586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.30.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.30.3
--webserver_port=0
--tserver_master_addrs=127.22.30.62:43811
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.30.3
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:07.953330 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:07.953567 21586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:07.958321 21599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.958359 21598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:07.958318 21601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:07.958456 21586 server_base.cc:1034] running on GCE node
I20260808 22:54:07.958696 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:07.958916 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:07.960050 21586 hybrid_clock.cc:648] HybridClock initialized: now 1786229647960026 us; error 40 us; skew 500 ppm
I20260808 22:54:07.961180 21586 webserver.cc:460] Webserver started at http://127.22.30.3:34567/ using document root <none> and password file <none>
I20260808 22:54:07.961392 21586 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:07.961436 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:07.961583 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:07.962505 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data/instance:
uuid: "40438c97801f4b24a4dcb0a0c8254682"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "3e0beba432f2b46a7bb37ac1ca8f9f16"
server_key_iv: "d304cf607857fe4d59af2f758604499a"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.962853 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance:
uuid: "40438c97801f4b24a4dcb0a0c8254682"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "3e0beba432f2b46a7bb37ac1ca8f9f16"
server_key_iv: "d304cf607857fe4d59af2f758604499a"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.964233 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:07.965015 21610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.965221 21586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:07.965322 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal
uuid: "40438c97801f4b24a4dcb0a0c8254682"
format_stamp: "Formatted at 2026-08-08 22:54:07 on dist-test-slave-x4qh"
server_key: "3e0beba432f2b46a7bb37ac1ca8f9f16"
server_key_iv: "d304cf607857fe4d59af2f758604499a"
server_key_version: "encryptionkey@0"
I20260808 22:54:07.965404 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:07.976452 21586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:07.976712 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:07.976855 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:07.977061 21586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 22:54:07.977380 21586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 22:54:07.977437 21586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.977473 21586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 22:54:07.977501 21586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:07.983546 21586 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.3:34495
I20260808 22:54:07.983601 21793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.3:34495 every 8 connection(s)
I20260808 22:54:07.983896 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/data/info.pb
I20260808 22:54:07.986625 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 21586
I20260808 22:54:07.986737 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal/instance
I20260808 22:54:07.986940 22648 external_mini_cluster.cc:1428] Setting key 1421c18e18d89e40519950ebe0a5b53c
I20260808 22:54:07.988864 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.984545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.3:58043 (local address 127.22.30.62:43811)
0808 22:54:07.984611 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.984613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.984817 (+ 204us) server_negotiation.cc:407] Connection header received
0808 22:54:07.985174 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.985175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.985186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.985241 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.985860 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.986335 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.987585 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.987755 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.987827 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.988598 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:07.988599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:07.988599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:07.988603 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:07.988618 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:07.988622 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:07.988668 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:07.988738 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 22:54:07.988763 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:54:07.989053 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.984541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:43811 (local address 127.22.30.3:58043)
0808 22:54:07.984715 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:07.984741 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:07.985100 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:07.985313 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:07.985316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:07.985323 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:07.985739 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.985742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.986416 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.986417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.987507 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:07.987513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:07.987839 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:07.987840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:07.987957 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:07.988456 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 22:54:07.988456 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:07.988480 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 22:54:07.988500 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:07.988667 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:07.988672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:07.988674 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:07.988739 (+ 65us) client_negotiation.cc:240] Negotiation successful
0808 22:54:07.988848 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260808 22:54:07.989496 21794 heartbeater.cc:347] Connected to a master server at 127.22.30.62:43811
I20260808 22:54:07.989557 21794 heartbeater.cc:464] Registering TS with master...
I20260808 22:54:07.989715 21794 heartbeater.cc:511] Master 127.22.30.62:43811 requested a full tablet report, sending...
I20260808 22:54:07.990020 21102 ts_manager.cc:194] Registered new tserver with Master: 40438c97801f4b24a4dcb0a0c8254682 (127.22.30.3:34495)
I20260808 22:54:07.990357 21102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.30.3:58043
I20260808 22:54:07.998457 22648 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 22:54:08.001853 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:07.999629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.22.30.62:43811)
0808 22:54:07.999674 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:07.999675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:07.999768 (+ 93us) server_negotiation.cc:407] Connection header received
0808 22:54:07.999824 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:07.999825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:07.999832 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:07.999863 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:07.999967 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.000372 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.001000 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.001162 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.001205 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.001655 (+ 450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:08.001655 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:08.001656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:08.001659 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:08.001669 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:08.001672 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:08.001712 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.001745 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.001759 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 22:54:08.003022 21102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 22:54:08.004467 21102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 22:54:08.010056 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.006328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:37261 (local address 127.0.0.1:41638)
0808 22:54:08.006460 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:08.006495 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:08.006552 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:08.007021 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:08.007021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:08.007023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:08.007081 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.007082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.008291 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.008292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.009063 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.009066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.009816 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.009817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.009884 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.009894 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 22:54:08.009953 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 22:54:08.009997 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":47,"thread_start_us":84,"threads_started":1}
I20260808 22:54:08.010066 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.006689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.22.30.1:37261)
0808 22:54:08.006876 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:08.006879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:08.006888 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:08.006926 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:08.006928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:08.006942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:08.007014 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:08.007173 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.008186 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.009235 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.009738 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.009813 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.009890 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.009963 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.009993 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260808 22:54:08.010264 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.006362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37873 (local address 127.0.0.1:42044)
0808 22:54:08.006642 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:08.006653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:08.006708 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:08.007009 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:08.007013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:08.007015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:08.007081 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.007082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.008330 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.008330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.009311 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.009313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.010076 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.010077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.010132 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.010135 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 22:54:08.010160 (+ 25us) client_negotiation.cc:240] Negotiation successful
0808 22:54:08.010192 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":6,"thread_start_us":83,"threads_started":1}
I20260808 22:54:08.010397 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.006546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:34495 (local address 127.0.0.1:51740)
0808 22:54:08.007074 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:08.007084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:08.007135 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:08.007414 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:08.007415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:08.007416 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:08.007451 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.007452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.008137 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.008138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.009293 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.009296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.010140 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.010141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.010187 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.010189 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 22:54:08.010228 (+ 39us) client_negotiation.cc:240] Negotiation successful
0808 22:54:08.010254 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":20,"thread_start_us":69,"threads_started":1}
I20260808 22:54:08.010409 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.006620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.22.30.3:34495)
0808 22:54:08.006798 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:08.006803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:08.007136 (+ 333us) server_negotiation.cc:407] Connection header received
0808 22:54:08.007203 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:08.007205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:08.007222 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:08.007328 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:08.007516 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.008057 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.009384 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.009984 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.010048 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.010114 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.010298 (+ 184us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.010335 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260808 22:54:08.010742 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.006478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.22.30.2:37873)
0808 22:54:08.006822 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:08.006827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:08.006836 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:08.006882 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:08.006884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:08.006901 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:08.007282 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:08.007355 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.008266 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.009387 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.009984 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.010508 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.010593 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.010612 (+ 19us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.010666 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":72,"threads_started":1}
I20260808 22:54:08.011142 21692 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb231c8bb8d4705903fddd12992fad2 (DEFAULT_TABLE table=test-table [id=a6c012cc1e9344d7bac07e481c1d54d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:08.011408 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb231c8bb8d4705903fddd12992fad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:08.011351 21278 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb231c8bb8d4705903fddd12992fad2 (DEFAULT_TABLE table=test-table [id=a6c012cc1e9344d7bac07e481c1d54d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:08.011350 21484 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb231c8bb8d4705903fddd12992fad2 (DEFAULT_TABLE table=test-table [id=a6c012cc1e9344d7bac07e481c1d54d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 22:54:08.011673 21484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb231c8bb8d4705903fddd12992fad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:08.011673 21278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb231c8bb8d4705903fddd12992fad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:08.013657 21819 tablet_bootstrap.cc:492] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Bootstrap starting.
I20260808 22:54:08.014219 21819 tablet_bootstrap.cc:654] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:08.014565 21819 log.cc:824] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:08.014905 21824 tablet_bootstrap.cc:492] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Bootstrap starting.
I20260808 22:54:08.015468 21819 tablet_bootstrap.cc:492] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: No bootstrap required, opened a new log
I20260808 22:54:08.015558 21824 tablet_bootstrap.cc:654] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:08.015548 21819 ts_tablet_manager.cc:1397] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:08.015805 21824 log.cc:824] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:08.016081 21822 tablet_bootstrap.cc:492] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005: Bootstrap starting.
I20260808 22:54:08.016613 21822 tablet_bootstrap.cc:654] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:08.016614 21824 tablet_bootstrap.cc:492] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: No bootstrap required, opened a new log
I20260808 22:54:08.016687 21824 ts_tablet_manager.cc:1397] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:08.016896 21822 log.cc:824] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:08.017524 21822 tablet_bootstrap.cc:492] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005: No bootstrap required, opened a new log
I20260808 22:54:08.017596 21822 ts_tablet_manager.cc:1397] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 22:54:08.018114 21819 raft_consensus.cc:348] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.018273 21819 raft_consensus.cc:374] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:08.018311 21819 raft_consensus.cc:729] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40438c97801f4b24a4dcb0a0c8254682, State: Initialized, Role: FOLLOWER
I20260808 22:54:08.018450 21819 consensus_queue.cc:260] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.018720 21819 ts_tablet_manager.cc:1428] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:08.018748 21794 heartbeater.cc:503] Master 127.22.30.62:43811 was elected leader, sending a full tablet report...
I20260808 22:54:08.019147 21824 raft_consensus.cc:348] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.019281 21824 raft_consensus.cc:374] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:08.019322 21824 raft_consensus.cc:729] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb37bb26c08745028abd8760b2d767ce, State: Initialized, Role: FOLLOWER
I20260808 22:54:08.019460 21824 consensus_queue.cc:260] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.019716 21824 ts_tablet_manager.cc:1428] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:08.019841 21583 heartbeater.cc:503] Master 127.22.30.62:43811 was elected leader, sending a full tablet report...
I20260808 22:54:08.020061 21822 raft_consensus.cc:348] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.020207 21822 raft_consensus.cc:374] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:08.020249 21822 raft_consensus.cc:729] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915b741547ca4766a3df84e82d80b005, State: Initialized, Role: FOLLOWER
I20260808 22:54:08.020368 21822 consensus_queue.cc:260] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.020604 21822 ts_tablet_manager.cc:1428] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 22:54:08.020640 21377 heartbeater.cc:503] Master 127.22.30.62:43811 was elected leader, sending a full tablet report...
I20260808 22:54:08.076828 21830 raft_consensus.cc:482] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 22:54:08.076924 21830 raft_consensus.cc:504] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.077260 21830 leader_election.cc:288] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb37bb26c08745028abd8760b2d767ce (127.22.30.2:37873), 40438c97801f4b24a4dcb0a0c8254682 (127.22.30.3:34495)
I20260808 22:54:08.080404 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.077696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:35307 (local address 127.22.30.3:34495)
0808 22:54:08.077746 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:08.077748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:08.077756 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:08.077790 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:08.077791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:08.077798 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:08.077855 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:08.078027 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.078447 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.079729 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.080060 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.080111 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.080117 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.080303 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.080345 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 22:54:08.080505 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.077599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.30.1:53701 (local address 127.22.30.2:37873)
0808 22:54:08.077636 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:08.077638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:08.077646 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:08.077676 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:08.077677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:08.077687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:08.077749 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 22:54:08.077905 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.078392 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.079720 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.080120 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.080193 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.080202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.080379 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.080417 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:54:08.080515 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.077420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.2:37873 (local address 127.22.30.1:53701)
0808 22:54:08.077470 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:08.077482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:08.077534 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:08.077767 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:08.077767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:08.077770 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:08.077838 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.077840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.078467 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.078468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.079618 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.079621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.080213 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.080214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.080310 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.080314 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:54:08.080365 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 22:54:08.080403 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":13}
I20260808 22:54:08.080406 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.077539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.3:34495 (local address 127.22.30.1:35307)
0808 22:54:08.077691 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:08.077701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:08.077746 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:08.077894 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:08.077895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:08.077897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 22:54:08.077965 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.077967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.078499 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.078499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.079619 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:08.079621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.080126 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.080128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:08.080215 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.080219 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 22:54:08.080288 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 22:54:08.080318 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":30,"thread_start_us":99,"threads_started":1}
I20260808 22:54:08.080804 21516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb231c8bb8d4705903fddd12992fad2" candidate_uuid: "915b741547ca4766a3df84e82d80b005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb37bb26c08745028abd8760b2d767ce" is_pre_election: true
I20260808 22:54:08.080928 21516 raft_consensus.cc:2454] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915b741547ca4766a3df84e82d80b005 in term 0.
I20260808 22:54:08.081174 21213 leader_election.cc:302] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915b741547ca4766a3df84e82d80b005, cb37bb26c08745028abd8760b2d767ce; no voters:
I20260808 22:54:08.081250 21723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb231c8bb8d4705903fddd12992fad2" candidate_uuid: "915b741547ca4766a3df84e82d80b005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40438c97801f4b24a4dcb0a0c8254682" is_pre_election: true
I20260808 22:54:08.081341 21830 raft_consensus.cc:2781] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 22:54:08.081382 21723 raft_consensus.cc:2454] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915b741547ca4766a3df84e82d80b005 in term 0.
I20260808 22:54:08.081408 21830 raft_consensus.cc:482] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 22:54:08.081450 21830 raft_consensus.cc:3037] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:08.082057 21830 raft_consensus.cc:504] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.082185 21830 leader_election.cc:288] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [CANDIDATE]: Term 1 election: Requested vote from peers cb37bb26c08745028abd8760b2d767ce (127.22.30.2:37873), 40438c97801f4b24a4dcb0a0c8254682 (127.22.30.3:34495)
I20260808 22:54:08.082300 21516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb231c8bb8d4705903fddd12992fad2" candidate_uuid: "915b741547ca4766a3df84e82d80b005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb37bb26c08745028abd8760b2d767ce"
I20260808 22:54:08.082312 21723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb231c8bb8d4705903fddd12992fad2" candidate_uuid: "915b741547ca4766a3df84e82d80b005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40438c97801f4b24a4dcb0a0c8254682"
I20260808 22:54:08.082362 21516 raft_consensus.cc:3037] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:08.082386 21723 raft_consensus.cc:3037] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:08.082996 21516 raft_consensus.cc:2454] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915b741547ca4766a3df84e82d80b005 in term 1.
I20260808 22:54:08.083046 21723 raft_consensus.cc:2454] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915b741547ca4766a3df84e82d80b005 in term 1.
I20260808 22:54:08.083202 21213 leader_election.cc:302] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915b741547ca4766a3df84e82d80b005, cb37bb26c08745028abd8760b2d767ce; no voters:
I20260808 22:54:08.083313 21830 raft_consensus.cc:2781] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:08.083494 21830 raft_consensus.cc:686] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [term 1 LEADER]: Becoming Leader. State: Replica: 915b741547ca4766a3df84e82d80b005, State: Running, Role: LEADER
I20260808 22:54:08.083590 21830 consensus_queue.cc:237] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } }
I20260808 22:54:08.084383 21102 catalog_manager.cc:5495] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 reported cstate change: term changed from 0 to 1, leader changed from <none> to 915b741547ca4766a3df84e82d80b005 (127.22.30.1). New cstate: current_term: 1 leader_uuid: "915b741547ca4766a3df84e82d80b005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915b741547ca4766a3df84e82d80b005" member_type: VOTER last_known_addr { host: "127.22.30.1" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 } health_report { overall_health: UNKNOWN } } }
I20260808 22:54:08.100865 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.098672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.22.30.1:37261)
0808 22:54:08.098723 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:08.098725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:08.098741 (+ 16us) server_negotiation.cc:407] Connection header received
0808 22:54:08.098792 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:08.098792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:08.098807 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:08.098871 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:08.098985 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.099443 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.100172 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:08.100352 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:08.100431 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:08.100572 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:08.100650 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:08.100730 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:08.100750 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 22:54:08.100767 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 22:54:08.102226 21516 raft_consensus.cc:1261] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 1 FOLLOWER]: Refusing update from remote peer 915b741547ca4766a3df84e82d80b005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:08.102242 21723 raft_consensus.cc:1261] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 1 FOLLOWER]: Refusing update from remote peer 915b741547ca4766a3df84e82d80b005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:08.102500 21832 consensus_queue.cc:1035] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb37bb26c08745028abd8760b2d767ce" member_type: VOTER last_known_addr { host: "127.22.30.2" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:08.102609 21830 consensus_queue.cc:1035] T 2bb231c8bb8d4705903fddd12992fad2 P 915b741547ca4766a3df84e82d80b005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40438c97801f4b24a4dcb0a0c8254682" member_type: VOTER last_known_addr { host: "127.22.30.3" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:08.105569 21836 mvcc.cc:205] Tried to move back new op lower bound from 7316396638624788480 to 7316396638551015424. Current Snapshot: MvccSnapshot[applied={T|T < 7316396638624788480}]
I20260808 22:54:08.106724 21102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:08.106820 21102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 22:54:08.107515 21102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eceaf1f70fd48d5852cecaa30dce1ae: Sending DeleteTablet for 3 replicas of tablet 2bb231c8bb8d4705903fddd12992fad2
I20260808 22:54:08.107805 21278 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb231c8bb8d4705903fddd12992fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:08 UTC) from {username='slave'} at 127.0.0.1:41638
I20260808 22:54:08.107825 21692 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb231c8bb8d4705903fddd12992fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:08 UTC) from {username='slave'} at 127.0.0.1:51740
I20260808 22:54:08.108007 21851 tablet_replica.cc:330] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: stopping tablet replica
I20260808 22:54:08.107998 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 21186
I20260808 22:54:08.108007 21484 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb231c8bb8d4705903fddd12992fad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 22:54:08 UTC) from {username='slave'} at 127.0.0.1:42044
I20260808 22:54:08.108090 21851 raft_consensus.cc:2229] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:08.108174 21851 raft_consensus.cc:2258] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:08.108893 21851 ts_tablet_manager.cc:1905] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:08.110670 21852 tablet_replica.cc:330] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: stopping tablet replica
I20260808 22:54:08.110774 21852 raft_consensus.cc:2229] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 22:54:08.110822 21851 ts_tablet_manager.cc:1918] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:08.110859 21852 raft_consensus.cc:2258] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 22:54:08.110908 21851 log.cc:1196] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-2/wal/wals/2bb231c8bb8d4705903fddd12992fad2
I20260808 22:54:08.111176 21851 ts_tablet_manager.cc:1939] T 2bb231c8bb8d4705903fddd12992fad2 P 40438c97801f4b24a4dcb0a0c8254682: Deleting consensus metadata
I20260808 22:54:08.112025 21083 catalog_manager.cc:4841] TS 40438c97801f4b24a4dcb0a0c8254682 (127.22.30.3:34495): tablet 2bb231c8bb8d4705903fddd12992fad2 (table test-table [id=a6c012cc1e9344d7bac07e481c1d54d7]) successfully deleted
I20260808 22:54:08.112954 21852 ts_tablet_manager.cc:1905] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 22:54:08.114058 21852 ts_tablet_manager.cc:1918] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 22:54:08.114142 21852 log.cc:1196] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Deleting WAL directory at /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786229410580610-22648-0/minicluster-data/ts-1/wal/wals/2bb231c8bb8d4705903fddd12992fad2
I20260808 22:54:08.114351 21852 ts_tablet_manager.cc:1939] T 2bb231c8bb8d4705903fddd12992fad2 P cb37bb26c08745028abd8760b2d767ce: Deleting consensus metadata
I20260808 22:54:08.114691 21083 catalog_manager.cc:4841] TS cb37bb26c08745028abd8760b2d767ce (127.22.30.2:37873): tablet 2bb231c8bb8d4705903fddd12992fad2 (table test-table [id=a6c012cc1e9344d7bac07e481c1d54d7]) successfully deleted
W20260808 22:54:08.123528 21084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.30.1:37261 (error 108)
I20260808 22:54:08.123765 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 21382
I20260808 22:54:08.123957 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:08.123852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.1:37261 (local address 127.0.0.1:41654)
0808 22:54:08.123897 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:08.123926 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.1:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 22:54:08.124051 21084 catalog_manager.cc:4584] TS 915b741547ca4766a3df84e82d80b005 (127.22.30.1:37261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bb231c8bb8d4705903fddd12992fad2: Network error: Client connection negotiation failed: client connection to 127.22.30.1:37261: connect: Connection refused (error 111)
I20260808 22:54:08.135699 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 21586
I20260808 22:54:08.143824 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 21058
2026-08-08T22:54:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (686 ms)
[----------] 6 tests from EncryptionPolicyTest (8445 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:08 dist-test-slave-x4qh krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:08 dist-test-slave-x4qh krb5kdc[21860](info): set up 2 sockets
Aug 08 22:54:08 dist-test-slave-x4qh krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260808 22:54:09.322944 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:09 dist-test-slave-x4qh krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:09.433951 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:09.512557 21881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:09.512773 21881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:09.512820 21881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:09.513831 21881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:09.513890 21881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:09.513914 21881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:09.513942 21881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:09.513967 21881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:09.515285 21881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:09.515555 21881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:09.515792 21881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:09.520542 21888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:09.520532 21890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:09.520515 21887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:09.520720 21881 server_base.cc:1034] running on GCE node
I20260808 22:54:09.520972 21881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:09.521243 21881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:09.522426 21881 hybrid_clock.cc:648] HybridClock initialized: now 1786229649522414 us; error 23 us; skew 500 ppm
Aug 08 22:54:09 dist-test-slave-x4qh krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:09.524736 21881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:09.525125 21881 webserver.cc:460] Webserver started at http://127.0.0.1:36613/ using document root <none> and password file <none>
I20260808 22:54:09.525343 21881 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:09.525411 21881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:09.525565 21881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:09.526515 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "22e7017c193d41a1b529abadea1729e1"
format_stamp: "Formatted at 2026-08-08 22:54:09 on dist-test-slave-x4qh"
server_key: "8258b612cb398631d9bf5753473dca14"
server_key_iv: "ff40b34ba05db04ca396408b4a4b5d32"
server_key_version: "encryptionkey@0"
I20260808 22:54:09.526867 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "22e7017c193d41a1b529abadea1729e1"
format_stamp: "Formatted at 2026-08-08 22:54:09 on dist-test-slave-x4qh"
server_key: "8258b612cb398631d9bf5753473dca14"
server_key_iv: "ff40b34ba05db04ca396408b4a4b5d32"
server_key_version: "encryptionkey@0"
I20260808 22:54:09.528301 21881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:09.529024 21899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:09.529187 21881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:09.529296 21881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "22e7017c193d41a1b529abadea1729e1"
format_stamp: "Formatted at 2026-08-08 22:54:09 on dist-test-slave-x4qh"
server_key: "8258b612cb398631d9bf5753473dca14"
server_key_iv: "ff40b34ba05db04ca396408b4a4b5d32"
server_key_version: "encryptionkey@0"
I20260808 22:54:09.529367 21881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:09.541309 21881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:09.542933 21881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:09.543085 21881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:09.547129 21881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46267
I20260808 22:54:09.547158 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46267 every 8 connection(s)
I20260808 22:54:09.547628 21881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:09.548022 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 21881
I20260808 22:54:09.548172 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:09.548413 22648 external_mini_cluster.cc:1428] Setting key a8729c38e113ac1bf3957d796d17e03e
I20260808 22:54:09.548673 21973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:09.551167 21973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1: Bootstrap starting.
I20260808 22:54:09.551944 21973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:09.552297 21973 log.cc:824] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:09.552973 21973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1: No bootstrap required, opened a new log
Aug 08 22:54:09 dist-test-slave-x4qh krb5kdc[21860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:54:09.555177 21973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e7017c193d41a1b529abadea1729e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260808 22:54:09.555307 21973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:09.555330 21973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e7017c193d41a1b529abadea1729e1, State: Initialized, Role: FOLLOWER
I20260808 22:54:09.555521 21973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e7017c193d41a1b529abadea1729e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260808 22:54:09.555586 21973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:09.555608 21973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:09.555678 21973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:09.556504 21973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e7017c193d41a1b529abadea1729e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260808 22:54:09.556622 21973 leader_election.cc:302] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e7017c193d41a1b529abadea1729e1; no voters:
I20260808 22:54:09.556807 21973 leader_election.cc:288] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:09.556892 21981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:09.557094 21973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:09.557117 21981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [term 1 LEADER]: Becoming Leader. State: Replica: 22e7017c193d41a1b529abadea1729e1, State: Running, Role: LEADER
I20260808 22:54:09.557206 21981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e7017c193d41a1b529abadea1729e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } }
I20260808 22:54:09.557533 21981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e7017c193d41a1b529abadea1729e1. Latest consensus state: current_term: 1 leader_uuid: "22e7017c193d41a1b529abadea1729e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e7017c193d41a1b529abadea1729e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } } }
I20260808 22:54:09.557628 21981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:09.557541 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e7017c193d41a1b529abadea1729e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e7017c193d41a1b529abadea1729e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46267 } } }
I20260808 22:54:09.557714 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:09.557920 21997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:09.558492 21997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:09.559767 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:09.548935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41783 (local address 127.0.0.1:46267)
0808 22:54:09.549121 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:09.549128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:09.549149 (+ 21us) server_negotiation.cc:407] Connection header received
0808 22:54:09.549617 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:09.549624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:09.549666 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:09.549794 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:09.550429 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:09.551237 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:09.552491 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:09.552965 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:09.553081 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:09.555708 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:09.555712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:09.555712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:09.555723 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:09.557909 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:09.558471 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:09.558475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:09.558475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:09.558584 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:09.558894 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:09.558895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:09.558895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:09.559124 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:09.559309 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:09.559432 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 22:54:09.559530 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260808 22:54:09.560686 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:09.564658 21997 catalog_manager.cc:1347] Generated new cluster ID: 10e35ca01ad54778a718acdd99067cb1
I20260808 22:54:09.564708 21997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:09.568656 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:09.561934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32961 (local address 127.0.0.1:46267)
0808 22:54:09.561964 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:09.561966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:09.562048 (+ 82us) server_negotiation.cc:407] Connection header received
0808 22:54:09.562115 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:09.562115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:09.562122 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:09.562163 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:09.562766 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:09.563195 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:09.564193 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:09.564411 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:09.564489 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:09.565793 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:09.565795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:09.565795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:09.565802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:09.567361 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:09.567790 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:09.567791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:09.567791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:09.567850 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:09.568067 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:09.568067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:09.568068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:09.568186 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:09.568305 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:09.568401 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 22:54:09.568461 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 22:54:09.590005 21997 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:09.590615 21997 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:09.594712 21997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22e7017c193d41a1b529abadea1729e1: Generated new TSK 0
I20260808 22:54:09.594852 21997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:09.603440 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 21881
W20260808 22:54:09.614169 21856 connection.cc:535] client connection to 127.0.0.1:46267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46267): Connection reset by peer (error 104)
2026-08-08T22:54:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:09 dist-test-slave-x4qh krb5kdc[22023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:09 dist-test-slave-x4qh krb5kdc[22023](info): set up 2 sockets
Aug 08 22:54:09 dist-test-slave-x4qh krb5kdc[22023](info): commencing operation
krb5kdc: starting...
W20260808 22:54:10.768219 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:10 dist-test-slave-x4qh krb5kdc[22023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:10.877971 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:10.953559 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:10.953732 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:10.953753 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:10.954794 22042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:10.954825 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:10.954856 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:10.954869 22042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:10.954880 22042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:10.956125 22042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:10.956306 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:10.956461 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:10.960991 22051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:10.961053 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:10.960999 22053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:10.961249 22042 server_base.cc:1034] running on GCE node
I20260808 22:54:10.961463 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:10.961735 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:10.962894 22042 hybrid_clock.cc:648] HybridClock initialized: now 1786229650962884 us; error 35 us; skew 500 ppm
Aug 08 22:54:10 dist-test-slave-x4qh krb5kdc[22023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:10.965262 22042 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:10.965665 22042 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
I20260808 22:54:10.965883 22042 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:10.965952 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:10.966101 22042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:10.967085 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "e7fba889895e4c569eaab2f39fb675b3"
format_stamp: "Formatted at 2026-08-08 22:54:10 on dist-test-slave-x4qh"
server_key: "fec364ee8f54e594ec3d3b43b7f17aeb"
server_key_iv: "d39e07dc761c09ed0e93f885e4e94b5c"
server_key_version: "encryptionkey@0"
I20260808 22:54:10.967494 22042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "e7fba889895e4c569eaab2f39fb675b3"
format_stamp: "Formatted at 2026-08-08 22:54:10 on dist-test-slave-x4qh"
server_key: "fec364ee8f54e594ec3d3b43b7f17aeb"
server_key_iv: "d39e07dc761c09ed0e93f885e4e94b5c"
server_key_version: "encryptionkey@0"
I20260808 22:54:10.968755 22042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:10.969415 22065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:10.969568 22042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:10.969663 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "e7fba889895e4c569eaab2f39fb675b3"
format_stamp: "Formatted at 2026-08-08 22:54:10 on dist-test-slave-x4qh"
server_key: "fec364ee8f54e594ec3d3b43b7f17aeb"
server_key_iv: "d39e07dc761c09ed0e93f885e4e94b5c"
server_key_version: "encryptionkey@0"
I20260808 22:54:10.969746 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:10.979342 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:10.980942 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:10.981105 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:10.984839 22042 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41633
I20260808 22:54:10.984891 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41633 every 8 connection(s)
I20260808 22:54:10.985215 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:10.986055 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:10.988161 22142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3: Bootstrap starting.
I20260808 22:54:10.988686 22142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:10.988956 22142 log.cc:824] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:10.989571 22142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3: No bootstrap required, opened a new log
I20260808 22:54:10.990969 22142 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fba889895e4c569eaab2f39fb675b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41633 } }
I20260808 22:54:10.991088 22142 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:10.991149 22142 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fba889895e4c569eaab2f39fb675b3, State: Initialized, Role: FOLLOWER
I20260808 22:54:10.991291 22142 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fba889895e4c569eaab2f39fb675b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41633 } }
I20260808 22:54:10.991351 22142 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:10.991413 22142 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:10.991462 22142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:10.992000 22142 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fba889895e4c569eaab2f39fb675b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41633 } }
I20260808 22:54:10.992120 22142 leader_election.cc:302] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7fba889895e4c569eaab2f39fb675b3; no voters:
I20260808 22:54:10.992142 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22042
I20260808 22:54:10.992244 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:10.992280 22142 leader_election.cc:288] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:10.992318 22145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:10.992497 22648 external_mini_cluster.cc:1428] Setting key d4e94ec4a57ecfbec61711699ddb50c1
I20260808 22:54:10.992506 22145 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [term 1 LEADER]: Becoming Leader. State: Replica: e7fba889895e4c569eaab2f39fb675b3, State: Running, Role: LEADER
I20260808 22:54:10.992542 22142 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:10.992611 22145 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fba889895e4c569eaab2f39fb675b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41633 } }
I20260808 22:54:10.993075 22146 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7fba889895e4c569eaab2f39fb675b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fba889895e4c569eaab2f39fb675b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41633 } } }
I20260808 22:54:10.993157 22146 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:10.993319 22147 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7fba889895e4c569eaab2f39fb675b3. Latest consensus state: current_term: 1 leader_uuid: "e7fba889895e4c569eaab2f39fb675b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7fba889895e4c569eaab2f39fb675b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41633 } } }
I20260808 22:54:10.993391 22147 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3 [sys.catalog]: This master's current role is: LEADER
W20260808 22:54:10.994388 22170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:10.994441 22170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:10.994491 22159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:10.994632 22159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:10.995991 22159 catalog_manager.cc:1347] Generated new cluster ID: 05ece741b7f44e61966ffd14dc15ce1d
I20260808 22:54:10.996047 22159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:10.998329 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:10.993118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42551 (local address 127.0.0.1:41633)
0808 22:54:10.993576 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:10.993585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:10.993617 (+ 32us) server_negotiation.cc:407] Connection header received
0808 22:54:10.993727 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:10.993733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:10.993791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:10.993924 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:10.994266 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:10.995295 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:10.996303 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:10.996917 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:10.997008 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:10.997719 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:10.997723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:10.997723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:10.997907 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:10.997940 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:10.997953 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:10.998025 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:10.998135 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 22:54:10.998156 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":86,"threads_started":1}
I20260808 22:54:10.999172 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:11.010601 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:11.003672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60887 (local address 127.0.0.1:41633)
0808 22:54:11.003720 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:11.003722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:11.007523 (+ 3801us) server_negotiation.cc:407] Connection header received
0808 22:54:11.007563 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:11.007564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:11.007575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:11.007638 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:11.007742 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:11.008419 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:11.009317 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:11.009629 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:11.009699 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:11.010357 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:11.010358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:11.010358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:11.010375 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:11.010390 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:11.010394 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:11.010454 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:11.010522 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 22:54:11.010534 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 22:54:11.020908 22159 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:11.021371 22159 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:11.026664 22159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7fba889895e4c569eaab2f39fb675b3: Generated new TSK 0
I20260808 22:54:11.026818 22159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:11.028947 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22042
W20260808 22:54:11.043051 22020 connection.cc:535] client connection to 127.0.0.1:41633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41633): Connection reset by peer (error 104)
2026-08-08T22:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:11 dist-test-slave-x4qh krb5kdc[22181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:11 dist-test-slave-x4qh krb5kdc[22181](info): set up 2 sockets
Aug 08 22:54:11 dist-test-slave-x4qh krb5kdc[22181](info): commencing operation
krb5kdc: starting...
W20260808 22:54:12.191910 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:12 dist-test-slave-x4qh krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:12.299201 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:12.374822 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:12.375036 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:12.375072 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:12.376202 22200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:12.376307 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:12.376344 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:12.376367 22200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:12.376392 22200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:12.377784 22200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:12.378026 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:12.378226 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:12.382620 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:12.382627 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:12.382763 22200 server_base.cc:1034] running on GCE node
W20260808 22:54:12.382634 22213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:12.383074 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:12.383303 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:12.384462 22200 hybrid_clock.cc:648] HybridClock initialized: now 1786229652384477 us; error 51 us; skew 500 ppm
Aug 08 22:54:12 dist-test-slave-x4qh krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:12.386847 22200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:12.387264 22200 webserver.cc:460] Webserver started at http://127.0.0.1:40411/ using document root <none> and password file <none>
I20260808 22:54:12.387504 22200 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:12.387547 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:12.387631 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:12.388528 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "f0bf33e944194b1f9740dfcc447d9be1"
format_stamp: "Formatted at 2026-08-08 22:54:12 on dist-test-slave-x4qh"
server_key: "795759a22ed243028bf704cb8efd7457"
server_key_iv: "181bf97edee083443babd94225769663"
server_key_version: "encryptionkey@0"
I20260808 22:54:12.388794 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "f0bf33e944194b1f9740dfcc447d9be1"
format_stamp: "Formatted at 2026-08-08 22:54:12 on dist-test-slave-x4qh"
server_key: "795759a22ed243028bf704cb8efd7457"
server_key_iv: "181bf97edee083443babd94225769663"
server_key_version: "encryptionkey@0"
I20260808 22:54:12.389915 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:54:12.390513 22224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:12.390683 22200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:12.390781 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "f0bf33e944194b1f9740dfcc447d9be1"
format_stamp: "Formatted at 2026-08-08 22:54:12 on dist-test-slave-x4qh"
server_key: "795759a22ed243028bf704cb8efd7457"
server_key_iv: "181bf97edee083443babd94225769663"
server_key_version: "encryptionkey@0"
I20260808 22:54:12.390833 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:12.393924 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:12.395540 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:12.395637 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:12.399315 22200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37951
I20260808 22:54:12.399365 22298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37951 every 8 connection(s)
I20260808 22:54:12.399677 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:12.400540 22299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:12.402479 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1: Bootstrap starting.
I20260808 22:54:12.402992 22299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:12.403287 22299 log.cc:824] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:12.403410 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22200
I20260808 22:54:12.403548 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:12.403749 22648 external_mini_cluster.cc:1428] Setting key 537d738804f86928a1dd2ee1a4d75e7d
I20260808 22:54:12.404071 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1: No bootstrap required, opened a new log
I20260808 22:54:12.405668 22299 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bf33e944194b1f9740dfcc447d9be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260808 22:54:12.405833 22299 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:12.405895 22299 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bf33e944194b1f9740dfcc447d9be1, State: Initialized, Role: FOLLOWER
I20260808 22:54:12.406028 22299 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bf33e944194b1f9740dfcc447d9be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260808 22:54:12.406128 22299 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:12.406165 22299 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:12.406263 22299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:12.406898 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:12.404185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58595 (local address 127.0.0.1:37951)
0808 22:54:12.404392 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:12.404401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:12.404453 (+ 52us) server_negotiation.cc:407] Connection header received
0808 22:54:12.405655 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:12.405661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:12.405685 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:12.405837 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:12.406493 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:12.406496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:12.406496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:12.406500 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:12.406545 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:12.406555 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:12.406626 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:12.406693 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 22:54:12.406730 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260808 22:54:12.407300 22299 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bf33e944194b1f9740dfcc447d9be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260808 22:54:12.407450 22299 leader_election.cc:302] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0bf33e944194b1f9740dfcc447d9be1; no voters:
I20260808 22:54:12.407608 22299 leader_election.cc:288] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:12.407635 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:12.407836 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [term 1 LEADER]: Becoming Leader. State: Replica: f0bf33e944194b1f9740dfcc447d9be1, State: Running, Role: LEADER
I20260808 22:54:12.407889 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:12.407894 22299 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:12.407987 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bf33e944194b1f9740dfcc447d9be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } }
I20260808 22:54:12.408335 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0bf33e944194b1f9740dfcc447d9be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bf33e944194b1f9740dfcc447d9be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } } }
I20260808 22:54:12.408412 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:12.408557 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0bf33e944194b1f9740dfcc447d9be1. Latest consensus state: current_term: 1 leader_uuid: "f0bf33e944194b1f9740dfcc447d9be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bf33e944194b1f9740dfcc447d9be1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37951 } } }
I20260808 22:54:12.408627 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1 [sys.catalog]: This master's current role is: LEADER
W20260808 22:54:12.409627 22342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:12.409715 22342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:12.409825 22319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:12.410015 22319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:12.410878 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:12.409579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50905 (local address 127.0.0.1:37951)
0808 22:54:12.409610 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:12.409612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:12.409865 (+ 253us) server_negotiation.cc:407] Connection header received
0808 22:54:12.409944 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:12.409945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:12.409961 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:12.410011 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:12.410618 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:12.410619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:12.410620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:12.410624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:12.410636 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:12.410641 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:12.410671 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:12.410807 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 22:54:12.410822 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:54:12.411559 22319 catalog_manager.cc:1347] Generated new cluster ID: 2350afd830cf48c492153dccdb8b36ae
I20260808 22:54:12.411615 22319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:12.419480 22319 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:12.419646 22319 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:12.422516 22319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0bf33e944194b1f9740dfcc447d9be1: Generated new TSK 0
I20260808 22:54:12.422679 22319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:12.425742 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22200
2026-08-08T22:54:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:12 dist-test-slave-x4qh krb5kdc[22350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:12 dist-test-slave-x4qh krb5kdc[22350](info): set up 2 sockets
Aug 08 22:54:12 dist-test-slave-x4qh krb5kdc[22350](info): commencing operation
krb5kdc: starting...
W20260808 22:54:13.598841 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:13 dist-test-slave-x4qh krb5kdc[22350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:13.707870 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:13.782644 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:13.782799 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:13.782819 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:13.783797 22370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:13.783833 22370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:13.783844 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:13.783855 22370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:13.783898 22370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:13.785173 22370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:13.785393 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:13.785589 22370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:13.789944 22377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:13.789984 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:13.789986 22376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:13.790133 22370 server_base.cc:1034] running on GCE node
I20260808 22:54:13.790417 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:13.790671 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:13.791805 22370 hybrid_clock.cc:648] HybridClock initialized: now 1786229653791797 us; error 31 us; skew 500 ppm
Aug 08 22:54:13 dist-test-slave-x4qh krb5kdc[22350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:13.793948 22370 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:13.794380 22370 webserver.cc:460] Webserver started at http://127.0.0.1:36875/ using document root <none> and password file <none>
I20260808 22:54:13.794587 22370 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:13.794629 22370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:13.794770 22370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:13.795781 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "8ada247a69e7412fae071a8d1914a7a6"
format_stamp: "Formatted at 2026-08-08 22:54:13 on dist-test-slave-x4qh"
server_key: "292282cbfa6135f78e0e48c6c134e591"
server_key_iv: "30c380babb1e2a9e0d8d43d5db1eb45e"
server_key_version: "encryptionkey@0"
I20260808 22:54:13.796114 22370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "8ada247a69e7412fae071a8d1914a7a6"
format_stamp: "Formatted at 2026-08-08 22:54:13 on dist-test-slave-x4qh"
server_key: "292282cbfa6135f78e0e48c6c134e591"
server_key_iv: "30c380babb1e2a9e0d8d43d5db1eb45e"
server_key_version: "encryptionkey@0"
I20260808 22:54:13.797374 22370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:13.798059 22396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:13.798218 22370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:13.798307 22370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "8ada247a69e7412fae071a8d1914a7a6"
format_stamp: "Formatted at 2026-08-08 22:54:13 on dist-test-slave-x4qh"
server_key: "292282cbfa6135f78e0e48c6c134e591"
server_key_iv: "30c380babb1e2a9e0d8d43d5db1eb45e"
server_key_version: "encryptionkey@0"
I20260808 22:54:13.798383 22370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:13.809448 22370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:13.811067 22370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:13.811216 22370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:13.814965 22370 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40267
I20260808 22:54:13.814993 22471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40267 every 8 connection(s)
I20260808 22:54:13.815277 22370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:13.816025 22472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:13.817991 22472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6: Bootstrap starting.
I20260808 22:54:13.818495 22472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:13.818763 22472 log.cc:824] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:13.819466 22472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6: No bootstrap required, opened a new log
I20260808 22:54:13.820868 22472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada247a69e7412fae071a8d1914a7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260808 22:54:13.820984 22472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:13.821005 22472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ada247a69e7412fae071a8d1914a7a6, State: Initialized, Role: FOLLOWER
I20260808 22:54:13.821178 22472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada247a69e7412fae071a8d1914a7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260808 22:54:13.821256 22472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:13.821301 22472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:13.821347 22472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:13.821815 22472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada247a69e7412fae071a8d1914a7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260808 22:54:13.821940 22472 leader_election.cc:302] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ada247a69e7412fae071a8d1914a7a6; no voters:
I20260808 22:54:13.822050 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22370
I20260808 22:54:13.822116 22472 leader_election.cc:288] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:13.822135 22475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:13.822177 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:13.822288 22475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8ada247a69e7412fae071a8d1914a7a6, State: Running, Role: LEADER
I20260808 22:54:13.822381 22648 external_mini_cluster.cc:1428] Setting key 0308a8e1d04b1fdda42462eceb1ecfbb
I20260808 22:54:13.822402 22472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:13.822408 22475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada247a69e7412fae071a8d1914a7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } }
I20260808 22:54:13.822729 22479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ada247a69e7412fae071a8d1914a7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada247a69e7412fae071a8d1914a7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } } }
I20260808 22:54:13.822818 22479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:13.822849 22477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ada247a69e7412fae071a8d1914a7a6. Latest consensus state: current_term: 1 leader_uuid: "8ada247a69e7412fae071a8d1914a7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ada247a69e7412fae071a8d1914a7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40267 } } }
I20260808 22:54:13.822922 22477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:13.823015 22486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:13.823532 22486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:13.825299 22486 catalog_manager.cc:1347] Generated new cluster ID: 8ae8fb42171d45a6bc3c181d7d2783f6
I20260808 22:54:13.825369 22486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:13 dist-test-slave-x4qh krb5kdc[22350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:54:13.830061 22486 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:13.830717 22486 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:13.833631 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:13.822929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55869 (local address 127.0.0.1:40267)
0808 22:54:13.823355 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:13.823382 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:13.823409 (+ 27us) server_negotiation.cc:407] Connection header received
0808 22:54:13.823514 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:13.823520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:13.823568 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:13.823684 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:13.824488 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:13.825778 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:13.826845 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:13.827497 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:13.827598 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:13.830299 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:13.830303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:13.830303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:13.830313 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:13.832138 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:13.832515 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:13.832518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:13.832519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:13.832611 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:13.832814 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:13.832815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:13.832815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:13.833035 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:13.833163 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:13.833274 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 22:54:13.833396 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":78,"threads_started":1}
I20260808 22:54:13.834826 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:13.835039 22486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ada247a69e7412fae071a8d1914a7a6: Generated new TSK 0
I20260808 22:54:13.835206 22486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:13.842118 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:13.836277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41401 (local address 127.0.0.1:40267)
0808 22:54:13.836310 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:13.836311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:13.836376 (+ 65us) server_negotiation.cc:407] Connection header received
0808 22:54:13.836438 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:13.836438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:13.836445 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:13.836481 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:13.837076 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:13.837573 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:13.838581 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:13.838757 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:13.838806 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:13.840035 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:13.840036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:13.840037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:13.840042 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:13.841189 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:13.841544 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:13.841544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:13.841545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:13.841587 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:13.841754 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:13.841754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:13.841755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:13.841837 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:13.841896 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:13.842022 (+ 126us) server_negotiation.cc:299] Negotiation successful
0808 22:54:13.842072 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:54:13.843868 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22370
2026-08-08T22:54:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:13 dist-test-slave-x4qh krb5kdc[22518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:13 dist-test-slave-x4qh krb5kdc[22518](info): set up 2 sockets
Aug 08 22:54:13 dist-test-slave-x4qh krb5kdc[22518](info): commencing operation
krb5kdc: starting...
W20260808 22:54:14.992038 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:15 dist-test-slave-x4qh krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:15.103703 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:15.184191 22535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:15.184386 22535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:15.184409 22535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:15.185500 22535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:15.185539 22535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:15.185554 22535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:15.185606 22535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:15.185628 22535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:15.187083 22535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:15.187316 22535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:15.187546 22535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:15.191932 22544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:15.191936 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:15.191951 22535 server_base.cc:1034] running on GCE node
W20260808 22:54:15.191974 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:15.192297 22535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:15.192559 22535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:15.193696 22535 hybrid_clock.cc:648] HybridClock initialized: now 1786229655193675 us; error 41 us; skew 500 ppm
Aug 08 22:54:15 dist-test-slave-x4qh krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:15.201574 22535 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:15.202008 22535 webserver.cc:460] Webserver started at http://127.0.0.1:36287/ using document root <none> and password file <none>
I20260808 22:54:15.202241 22535 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:15.202304 22535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:15.202462 22535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:15.203482 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "f5935fe713be4e8aaa291d86166fec19"
format_stamp: "Formatted at 2026-08-08 22:54:15 on dist-test-slave-x4qh"
server_key: "29bc4182a70e34978d9791401525848c"
server_key_iv: "5f1cda93938329d1bc0a1cbed071a0b9"
server_key_version: "encryptionkey@0"
I20260808 22:54:15.203817 22535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "f5935fe713be4e8aaa291d86166fec19"
format_stamp: "Formatted at 2026-08-08 22:54:15 on dist-test-slave-x4qh"
server_key: "29bc4182a70e34978d9791401525848c"
server_key_iv: "5f1cda93938329d1bc0a1cbed071a0b9"
server_key_version: "encryptionkey@0"
I20260808 22:54:15.205057 22535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:15.205720 22553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:15.205878 22535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:15.205976 22535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "f5935fe713be4e8aaa291d86166fec19"
format_stamp: "Formatted at 2026-08-08 22:54:15 on dist-test-slave-x4qh"
server_key: "29bc4182a70e34978d9791401525848c"
server_key_iv: "5f1cda93938329d1bc0a1cbed071a0b9"
server_key_version: "encryptionkey@0"
I20260808 22:54:15.206054 22535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:15.217187 22535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:15.218756 22535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:15.218894 22535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:15.222777 22535 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35833
I20260808 22:54:15.222805 22630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35833 every 8 connection(s)
I20260808 22:54:15.223102 22535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:15.223975 22631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:15.225940 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19: Bootstrap starting.
I20260808 22:54:15.226385 22631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:15.226624 22631 log.cc:824] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:15.227262 22631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19: No bootstrap required, opened a new log
I20260808 22:54:15.227763 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22535
I20260808 22:54:15.227898 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:15.228088 22648 external_mini_cluster.cc:1428] Setting key 03966ba88d241ebda7bdbb6a3f0faea6
I20260808 22:54:15.229310 22631 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5935fe713be4e8aaa291d86166fec19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260808 22:54:15.229455 22631 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:15.229487 22631 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5935fe713be4e8aaa291d86166fec19, State: Initialized, Role: FOLLOWER
I20260808 22:54:15.229638 22631 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5935fe713be4e8aaa291d86166fec19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260808 22:54:15.229715 22631 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:15.229745 22631 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:15.229802 22631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:15.230420 22631 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5935fe713be4e8aaa291d86166fec19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260808 22:54:15.230511 22631 leader_election.cc:302] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5935fe713be4e8aaa291d86166fec19; no voters:
I20260808 22:54:15.230635 22631 leader_election.cc:288] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:15.230695 22639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:15.230834 22631 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:15.230856 22639 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [term 1 LEADER]: Becoming Leader. State: Replica: f5935fe713be4e8aaa291d86166fec19, State: Running, Role: LEADER
I20260808 22:54:15.231072 22639 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5935fe713be4e8aaa291d86166fec19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260808 22:54:15.231480 22640 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5935fe713be4e8aaa291d86166fec19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5935fe713be4e8aaa291d86166fec19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } } }
I20260808 22:54:15.231564 22640 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:15.231463 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5935fe713be4e8aaa291d86166fec19. Latest consensus state: current_term: 1 leader_uuid: "f5935fe713be4e8aaa291d86166fec19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5935fe713be4e8aaa291d86166fec19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } } }
I20260808 22:54:15.231858 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:15.231776 22659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:15.232352 22659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:15.233851 22659 catalog_manager.cc:1347] Generated new cluster ID: 295a1f5d980b42b88a1ff53d291cde24
I20260808 22:54:15.233899 22659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:15.234344 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:15.228603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54589 (local address 127.0.0.1:35833)
0808 22:54:15.228786 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:15.228794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:15.228829 (+ 35us) server_negotiation.cc:407] Connection header received
0808 22:54:15.229288 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:15.229296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:15.229350 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:15.229480 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:15.229676 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:15.231159 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:15.232327 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:15.232824 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:15.232918 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:15.233675 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:15.233678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:15.233679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:15.233884 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:15.233917 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:15.233930 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:15.233995 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:15.234101 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 22:54:15.234163 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260808 22:54:15.235141 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:15.239676 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:15.236545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51153 (local address 127.0.0.1:35833)
0808 22:54:15.236574 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:15.236576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:15.236683 (+ 107us) server_negotiation.cc:407] Connection header received
0808 22:54:15.236723 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:15.236724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:15.236744 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:15.236819 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:15.236936 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:15.237635 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:15.238553 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:15.238844 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:15.238922 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:15.239466 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:15.239467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:15.239468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:15.239471 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:15.239484 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:15.239488 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:15.239549 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:15.239592 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 22:54:15.239613 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 22:54:15.248440 22659 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:15.249046 22659 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:15.252969 22659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5935fe713be4e8aaa291d86166fec19: Generated new TSK 0
I20260808 22:54:15.253136 22659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:15.255968 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22535
2026-08-08T22:54:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:15 dist-test-slave-x4qh krb5kdc[22683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:15 dist-test-slave-x4qh krb5kdc[22683](info): set up 2 sockets
Aug 08 22:54:15 dist-test-slave-x4qh krb5kdc[22683](info): commencing operation
krb5kdc: starting...
W20260808 22:54:16.427902 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:16 dist-test-slave-x4qh krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:16.535184 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:16.608338 22703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:16.608502 22703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:16.608523 22703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:16.609587 22703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:16.609624 22703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:16.609638 22703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:16.609649 22703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:16.609699 22703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:16.610960 22703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:16.611166 22703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:16.611384 22703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:16.615666 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:16.615665 22709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:16.615712 22710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:16.615826 22703 server_base.cc:1034] running on GCE node
I20260808 22:54:16.616089 22703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:16.616343 22703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:16.617476 22703 hybrid_clock.cc:648] HybridClock initialized: now 1786229656617468 us; error 25 us; skew 500 ppm
Aug 08 22:54:16 dist-test-slave-x4qh krb5kdc[22683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:16.619625 22703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:16.620045 22703 webserver.cc:460] Webserver started at http://127.0.0.1:46615/ using document root <none> and password file <none>
I20260808 22:54:16.620239 22703 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:16.620313 22703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:16.620443 22703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:16.621270 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "3ede98bb791d422984fc52fc181c5f69"
format_stamp: "Formatted at 2026-08-08 22:54:16 on dist-test-slave-x4qh"
server_key: "862eb683d0af4713996e9de984da9d3c"
server_key_iv: "e15b15dbbc1d08f6567ae0946ddb5089"
server_key_version: "encryptionkey@0"
I20260808 22:54:16.621573 22703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "3ede98bb791d422984fc52fc181c5f69"
format_stamp: "Formatted at 2026-08-08 22:54:16 on dist-test-slave-x4qh"
server_key: "862eb683d0af4713996e9de984da9d3c"
server_key_iv: "e15b15dbbc1d08f6567ae0946ddb5089"
server_key_version: "encryptionkey@0"
I20260808 22:54:16.622696 22703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:54:16.623333 22722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:16.623509 22703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:16.623598 22703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "3ede98bb791d422984fc52fc181c5f69"
format_stamp: "Formatted at 2026-08-08 22:54:16 on dist-test-slave-x4qh"
server_key: "862eb683d0af4713996e9de984da9d3c"
server_key_iv: "e15b15dbbc1d08f6567ae0946ddb5089"
server_key_version: "encryptionkey@0"
I20260808 22:54:16.623673 22703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:16.626443 22703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:16.628124 22703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:16.628257 22703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:16.631989 22703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38103
I20260808 22:54:16.632035 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38103 every 8 connection(s)
I20260808 22:54:16.632324 22703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:16.633131 22796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:16.634917 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69: Bootstrap starting.
I20260808 22:54:16.635485 22796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:16.635744 22796 log.cc:824] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:16.636336 22796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69: No bootstrap required, opened a new log
I20260808 22:54:16.637737 22796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ede98bb791d422984fc52fc181c5f69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38103 } }
I20260808 22:54:16.637861 22796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:16.637883 22796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ede98bb791d422984fc52fc181c5f69, State: Initialized, Role: FOLLOWER
I20260808 22:54:16.638055 22796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ede98bb791d422984fc52fc181c5f69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38103 } }
I20260808 22:54:16.638125 22796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:16.638145 22796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:16.638223 22796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:16.638692 22796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ede98bb791d422984fc52fc181c5f69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38103 } }
I20260808 22:54:16.638815 22796 leader_election.cc:302] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ede98bb791d422984fc52fc181c5f69; no voters:
I20260808 22:54:16.638983 22796 leader_election.cc:288] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:16.639030 22800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:16.639192 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22703
I20260808 22:54:16.639213 22800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [term 1 LEADER]: Becoming Leader. State: Replica: 3ede98bb791d422984fc52fc181c5f69, State: Running, Role: LEADER
I20260808 22:54:16.639243 22796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:16.639299 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:16.639294 22800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ede98bb791d422984fc52fc181c5f69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38103 } }
I20260808 22:54:16.639528 22648 external_mini_cluster.cc:1428] Setting key ac049ca9fa856d39b344b7c3aef0b716
I20260808 22:54:16.639621 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ede98bb791d422984fc52fc181c5f69. Latest consensus state: current_term: 1 leader_uuid: "3ede98bb791d422984fc52fc181c5f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ede98bb791d422984fc52fc181c5f69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38103 } } }
I20260808 22:54:16.639693 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:16.639743 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ede98bb791d422984fc52fc181c5f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ede98bb791d422984fc52fc181c5f69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38103 } } }
I20260808 22:54:16.639818 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:16.640046 22807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:16.641105 22807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:16.642550 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:16.639960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52441 (local address 127.0.0.1:38103)
0808 22:54:16.640173 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:16.640182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:16.640770 (+ 588us) server_negotiation.cc:407] Connection header received
0808 22:54:16.640996 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:16.641002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:16.641029 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:16.641158 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:16.642123 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:16.642127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:16.642127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:16.642131 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:16.642174 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:16.642187 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:16.642249 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:16.642327 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 22:54:16.642363 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260808 22:54:16.642553 22807 catalog_manager.cc:1347] Generated new cluster ID: 53db8e50e85745b0912f68aed5aeee6c
I20260808 22:54:16.642614 22807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:16.643347 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:16.645659 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:16.644654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50017 (local address 127.0.0.1:38103)
0808 22:54:16.644679 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:16.644681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:16.644842 (+ 161us) server_negotiation.cc:407] Connection header received
0808 22:54:16.644898 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:16.644898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:16.644902 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:16.644954 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:16.645461 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:16.645461 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:16.645462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:16.645465 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:16.645476 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:16.645480 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:16.645542 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:16.645600 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 22:54:16.645610 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260808 22:54:16.658493 22807 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:16.658631 22807 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:16.661141 22807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ede98bb791d422984fc52fc181c5f69: Generated new TSK 0
I20260808 22:54:16.661293 22807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:16.667523 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22703
2026-08-08T22:54:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:16 dist-test-slave-x4qh krb5kdc[22839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:16 dist-test-slave-x4qh krb5kdc[22839](info): set up 2 sockets
Aug 08 22:54:16 dist-test-slave-x4qh krb5kdc[22839](info): commencing operation
krb5kdc: starting...
W20260808 22:54:17.837086 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:17 dist-test-slave-x4qh krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:17.943285 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:18.020181 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:18.020413 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:18.020449 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:18.021481 22855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:18.021543 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:18.021565 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:18.021585 22855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:18.021610 22855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:18.022979 22855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:18.023243 22855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:18.023471 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:18.027858 22861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:18.027871 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:18.027871 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:18.028137 22855 server_base.cc:1034] running on GCE node
I20260808 22:54:18.028368 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:18.028625 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:18.029783 22855 hybrid_clock.cc:648] HybridClock initialized: now 1786229658029769 us; error 40 us; skew 500 ppm
Aug 08 22:54:18 dist-test-slave-x4qh krb5kdc[22839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:18.032032 22855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:18.032383 22855 webserver.cc:460] Webserver started at http://127.0.0.1:33097/ using document root <none> and password file <none>
I20260808 22:54:18.032605 22855 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:18.032666 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:18.032806 22855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:18.033677 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "7286c838b21c4e10817d4dc643dd72cd"
format_stamp: "Formatted at 2026-08-08 22:54:18 on dist-test-slave-x4qh"
server_key: "3988afcd429dc38ee523eece8410bc4b"
server_key_iv: "d982e2ba87b67d9d8a8bd1f1f913dbf9"
server_key_version: "encryptionkey@0"
I20260808 22:54:18.034024 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "7286c838b21c4e10817d4dc643dd72cd"
format_stamp: "Formatted at 2026-08-08 22:54:18 on dist-test-slave-x4qh"
server_key: "3988afcd429dc38ee523eece8410bc4b"
server_key_iv: "d982e2ba87b67d9d8a8bd1f1f913dbf9"
server_key_version: "encryptionkey@0"
I20260808 22:54:18.035244 22855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:18.035989 22873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:18.036150 22855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:18.036239 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "7286c838b21c4e10817d4dc643dd72cd"
format_stamp: "Formatted at 2026-08-08 22:54:18 on dist-test-slave-x4qh"
server_key: "3988afcd429dc38ee523eece8410bc4b"
server_key_iv: "d982e2ba87b67d9d8a8bd1f1f913dbf9"
server_key_version: "encryptionkey@0"
I20260808 22:54:18.036316 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:18.058714 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:18.060851 22855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:18.060985 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:18.064618 22855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40067
I20260808 22:54:18.064647 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40067 every 8 connection(s)
I20260808 22:54:18.064939 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:18.065702 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:18.067576 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 22855
I20260808 22:54:18.067701 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:18.067857 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd: Bootstrap starting.
I20260808 22:54:18.067903 22648 external_mini_cluster.cc:1428] Setting key 13a285e768b7e9a4cf09c4e4ae3a9661
I20260808 22:54:18.068524 22949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:18.068859 22949 log.cc:824] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:18.069640 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd: No bootstrap required, opened a new log
I20260808 22:54:18.071494 22949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7286c838b21c4e10817d4dc643dd72cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260808 22:54:18.071626 22949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:18.071655 22949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7286c838b21c4e10817d4dc643dd72cd, State: Initialized, Role: FOLLOWER
I20260808 22:54:18.071755 22949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7286c838b21c4e10817d4dc643dd72cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260808 22:54:18.071835 22949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:18.071897 22949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:18.071939 22949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:18.072465 22949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7286c838b21c4e10817d4dc643dd72cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260808 22:54:18.072571 22949 leader_election.cc:302] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7286c838b21c4e10817d4dc643dd72cd; no voters:
I20260808 22:54:18.072736 22949 leader_election.cc:288] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:18.072777 22955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:18.072935 22955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [term 1 LEADER]: Becoming Leader. State: Replica: 7286c838b21c4e10817d4dc643dd72cd, State: Running, Role: LEADER
I20260808 22:54:18.073006 22955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7286c838b21c4e10817d4dc643dd72cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260808 22:54:18.073024 22949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:18.073253 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7286c838b21c4e10817d4dc643dd72cd. Latest consensus state: current_term: 1 leader_uuid: "7286c838b21c4e10817d4dc643dd72cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7286c838b21c4e10817d4dc643dd72cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } } }
I20260808 22:54:18.073262 22956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7286c838b21c4e10817d4dc643dd72cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7286c838b21c4e10817d4dc643dd72cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } } }
I20260808 22:54:18.073344 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:18.073366 22956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd [sys.catalog]: This master's current role is: LEADER
Aug 08 22:54:18 dist-test-slave-x4qh krb5kdc[22839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260808 22:54:18.075433 22974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:18.075655 22974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:18.075764 22962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:18.076157 22962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:18.077637 22962 catalog_manager.cc:1347] Generated new cluster ID: afe2f1d71f654957b73825851956fd0a
I20260808 22:54:18.077687 22962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:18.078672 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:18.068485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:48341 (local address 127.0.0.1:40067)
0808 22:54:18.068641 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:18.068648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:18.068671 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:54:18.069167 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:18.069175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:18.069234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:18.069369 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:18.070100 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:18.070840 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:18.071851 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:18.072263 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:18.072331 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:18.075010 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:18.075013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:18.075014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:18.075023 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:18.076834 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:18.077483 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:18.077486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:18.077486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:18.077589 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:18.077869 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:18.077871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:18.077871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:18.078141 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:18.078262 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:18.078413 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 22:54:18.078507 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260808 22:54:18.079514 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:18.086890 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:18.080738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:53681 (local address 127.0.0.1:40067)
0808 22:54:18.080779 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:18.080781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:18.080847 (+ 66us) server_negotiation.cc:407] Connection header received
0808 22:54:18.080914 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:18.080915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:18.080921 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:18.080942 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:18.081456 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:18.082048 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:18.082837 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:18.083092 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:18.083121 (+ 29us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:18.084464 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:18.084465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:18.084465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:18.084471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:18.085863 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:18.086202 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:18.086202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:18.086203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:18.086241 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:18.086424 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:18.086425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:18.086425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:18.086511 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:18.086562 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:18.086741 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 22:54:18.086820 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:54:18.095525 22962 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:18.095999 22962 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:18.101577 22962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7286c838b21c4e10817d4dc643dd72cd: Generated new TSK 0
I20260808 22:54:18.101720 22962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:18.105775 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 22855
W20260808 22:54:18.110540 22837 connection.cc:535] client connection to 127.0.0.1:40067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40067): Connection reset by peer (error 104)
2026-08-08T22:54:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:18 dist-test-slave-x4qh krb5kdc[22989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:18 dist-test-slave-x4qh krb5kdc[22989](info): set up 2 sockets
Aug 08 22:54:18 dist-test-slave-x4qh krb5kdc[22989](info): commencing operation
krb5kdc: starting...
W20260808 22:54:19.251307 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:19 dist-test-slave-x4qh krb5kdc[22989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:19.358150 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:19.434396 23007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:19.434582 23007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:19.434602 23007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:19.435700 23007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:19.435741 23007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:19.435756 23007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:19.435768 23007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:19.435820 23007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:19.437227 23007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:19.437458 23007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:19.437669 23007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:19.442178 23016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:19.442157 23014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:19.442239 23007 server_base.cc:1034] running on GCE node
W20260808 22:54:19.442168 23013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:19.442674 23007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:19.442948 23007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:19.444078 23007 hybrid_clock.cc:648] HybridClock initialized: now 1786229659444068 us; error 25 us; skew 500 ppm
Aug 08 22:54:19 dist-test-slave-x4qh krb5kdc[22989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:19.446261 23007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:19.446663 23007 webserver.cc:460] Webserver started at http://127.0.0.1:38043/ using document root <none> and password file <none>
I20260808 22:54:19.446830 23007 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:19.446866 23007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:19.446943 23007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:19.447933 23007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "556c680d36a142a99d5f6d0dda83be3b"
format_stamp: "Formatted at 2026-08-08 22:54:19 on dist-test-slave-x4qh"
server_key: "33832c8095fb056d386675c1ad3fb6ef"
server_key_iv: "e30e7f957eed687433871f6e9d8bd427"
server_key_version: "encryptionkey@0"
I20260808 22:54:19.448220 23007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "556c680d36a142a99d5f6d0dda83be3b"
format_stamp: "Formatted at 2026-08-08 22:54:19 on dist-test-slave-x4qh"
server_key: "33832c8095fb056d386675c1ad3fb6ef"
server_key_iv: "e30e7f957eed687433871f6e9d8bd427"
server_key_version: "encryptionkey@0"
I20260808 22:54:19.449400 23007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:19.450018 23024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:19.450174 23007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:19.450274 23007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "556c680d36a142a99d5f6d0dda83be3b"
format_stamp: "Formatted at 2026-08-08 22:54:19 on dist-test-slave-x4qh"
server_key: "33832c8095fb056d386675c1ad3fb6ef"
server_key_iv: "e30e7f957eed687433871f6e9d8bd427"
server_key_version: "encryptionkey@0"
I20260808 22:54:19.450344 23007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:19.462016 23007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:19.463693 23007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:19.463838 23007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:19.467710 23007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41141
I20260808 22:54:19.467741 23099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41141 every 8 connection(s)
I20260808 22:54:19.468055 23007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:19.468822 23100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:19.470886 23100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b: Bootstrap starting.
I20260808 22:54:19.471449 23100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:19.471724 23100 log.cc:824] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:19.472417 23100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b: No bootstrap required, opened a new log
I20260808 22:54:19.472409 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23007
I20260808 22:54:19.472519 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:19.472719 22648 external_mini_cluster.cc:1428] Setting key 19a906aabfd12f47124c5feb87159cc5
I20260808 22:54:19.474365 23100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c680d36a142a99d5f6d0dda83be3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260808 22:54:19.474514 23100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:19.474550 23100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c680d36a142a99d5f6d0dda83be3b, State: Initialized, Role: FOLLOWER
I20260808 22:54:19.474727 23100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c680d36a142a99d5f6d0dda83be3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260808 22:54:19.474815 23100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:19.474838 23100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:19.474902 23100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:19.475449 23100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c680d36a142a99d5f6d0dda83be3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260808 22:54:19.475526 23100 leader_election.cc:302] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556c680d36a142a99d5f6d0dda83be3b; no voters:
I20260808 22:54:19.475703 23100 leader_election.cc:288] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:19.475726 23105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:19.475835 23105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [term 1 LEADER]: Becoming Leader. State: Replica: 556c680d36a142a99d5f6d0dda83be3b, State: Running, Role: LEADER
I20260808 22:54:19.475941 23100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:19.475961 23105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c680d36a142a99d5f6d0dda83be3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } }
I20260808 22:54:19.476310 23105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556c680d36a142a99d5f6d0dda83be3b. Latest consensus state: current_term: 1 leader_uuid: "556c680d36a142a99d5f6d0dda83be3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c680d36a142a99d5f6d0dda83be3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } } }
I20260808 22:54:19.476325 23106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556c680d36a142a99d5f6d0dda83be3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556c680d36a142a99d5f6d0dda83be3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41141 } } }
I20260808 22:54:19.476521 23106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:19.476701 23105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:19.477022 23116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:19.477603 23116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:19.479033 23116 catalog_manager.cc:1347] Generated new cluster ID: c5287a63b3ba4eeab1a3f2baf88cce47
I20260808 22:54:19.479076 23116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:19.479525 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:19.473237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:48049 (local address 127.0.0.1:41141)
0808 22:54:19.473438 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:19.473450 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:19.473490 (+ 40us) server_negotiation.cc:407] Connection header received
0808 22:54:19.474351 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:19.474357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:19.474400 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:19.474502 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:19.474779 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:19.476213 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:19.477306 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:19.478019 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:19.478136 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:19.478822 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:19.478825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:19.478825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:19.479042 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:19.479081 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:19.479091 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:19.479176 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:19.479252 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 22:54:19.479277 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260808 22:54:19.480376 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:19.490190 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:19.481592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:47859 (local address 127.0.0.1:41141)
0808 22:54:19.481623 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:19.481625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:19.487622 (+ 5997us) server_negotiation.cc:407] Connection header received
0808 22:54:19.487693 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:19.487693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:19.487704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:19.487758 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:19.487895 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:19.488361 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:19.489039 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:19.489245 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:19.489336 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:19.489945 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:19.489946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:19.489946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:19.489950 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:19.489962 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:19.489966 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:19.490021 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:19.490090 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 22:54:19.490102 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 22:54:19.491196 23116 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:19.491887 23116 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:19.499531 23116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556c680d36a142a99d5f6d0dda83be3b: Generated new TSK 0
I20260808 22:54:19.499681 23116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:19.502935 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23007
W20260808 22:54:19.526731 22986 connection.cc:535] client connection to 127.0.0.1:41141 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41141): Connection reset by peer (error 104)
2026-08-08T22:54:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:19 dist-test-slave-x4qh krb5kdc[23145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:19 dist-test-slave-x4qh krb5kdc[23145](info): set up 2 sockets
Aug 08 22:54:19 dist-test-slave-x4qh krb5kdc[23145](info): commencing operation
krb5kdc: starting...
W20260808 22:54:20.677557 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:20 dist-test-slave-x4qh krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:20.785854 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:20.860657 23165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:20.860852 23165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:20.860873 23165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:20.861836 23165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:20.861871 23165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:20.861884 23165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:20.861915 23165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:20.861941 23165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:20.863147 23165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:20.863306 23165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:20.863523 23165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:20.868026 23172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:20.868045 23171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:20.868069 23165 server_base.cc:1034] running on GCE node
W20260808 22:54:20.868045 23174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:20.868498 23165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:20.868726 23165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:20.869851 23165 hybrid_clock.cc:648] HybridClock initialized: now 1786229660869843 us; error 30 us; skew 500 ppm
Aug 08 22:54:20 dist-test-slave-x4qh krb5kdc[23145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:20.875648 23165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:20.876116 23165 webserver.cc:460] Webserver started at http://127.0.0.1:46163/ using document root <none> and password file <none>
I20260808 22:54:20.876338 23165 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:20.876381 23165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:20.876668 23165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:20.877671 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "ea65145557a642ea8a02453d2b791f41"
format_stamp: "Formatted at 2026-08-08 22:54:20 on dist-test-slave-x4qh"
server_key: "cc7e6d0be357b18d3378e082442cb96e"
server_key_iv: "f992c74e46cd54df7761e49a9efc3bb1"
server_key_version: "encryptionkey@0"
I20260808 22:54:20.878017 23165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "ea65145557a642ea8a02453d2b791f41"
format_stamp: "Formatted at 2026-08-08 22:54:20 on dist-test-slave-x4qh"
server_key: "cc7e6d0be357b18d3378e082442cb96e"
server_key_iv: "f992c74e46cd54df7761e49a9efc3bb1"
server_key_version: "encryptionkey@0"
I20260808 22:54:20.879254 23165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:20.880000 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:20.880208 23165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:20.880324 23165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "ea65145557a642ea8a02453d2b791f41"
format_stamp: "Formatted at 2026-08-08 22:54:20 on dist-test-slave-x4qh"
server_key: "cc7e6d0be357b18d3378e082442cb96e"
server_key_iv: "f992c74e46cd54df7761e49a9efc3bb1"
server_key_version: "encryptionkey@0"
I20260808 22:54:20.880427 23165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:20.883570 23165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:20.885442 23165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:20.885604 23165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:20.889791 23165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39547
I20260808 22:54:20.889828 23261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39547 every 8 connection(s)
I20260808 22:54:20.890122 23165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:20.890928 23262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:20.892858 23262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41: Bootstrap starting.
I20260808 22:54:20.893342 23262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:20.893601 23262 log.cc:824] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:20.894235 23262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41: No bootstrap required, opened a new log
I20260808 22:54:20.896281 23262 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea65145557a642ea8a02453d2b791f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39547 } }
I20260808 22:54:20.896394 23262 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:20.896430 23262 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea65145557a642ea8a02453d2b791f41, State: Initialized, Role: FOLLOWER
I20260808 22:54:20.896555 23262 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea65145557a642ea8a02453d2b791f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39547 } }
I20260808 22:54:20.896636 23262 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:20.896684 23262 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:20.896728 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:20.897214 23262 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea65145557a642ea8a02453d2b791f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39547 } }
I20260808 22:54:20.897337 23262 leader_election.cc:302] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea65145557a642ea8a02453d2b791f41; no voters:
I20260808 22:54:20.897483 23262 leader_election.cc:288] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:20.897518 23267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:20.897708 23267 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [term 1 LEADER]: Becoming Leader. State: Replica: ea65145557a642ea8a02453d2b791f41, State: Running, Role: LEADER
I20260808 22:54:20.897712 23262 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:20.897823 23267 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea65145557a642ea8a02453d2b791f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39547 } }
I20260808 22:54:20.898214 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea65145557a642ea8a02453d2b791f41. Latest consensus state: current_term: 1 leader_uuid: "ea65145557a642ea8a02453d2b791f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea65145557a642ea8a02453d2b791f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39547 } } }
I20260808 22:54:20.898211 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea65145557a642ea8a02453d2b791f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea65145557a642ea8a02453d2b791f41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39547 } } }
I20260808 22:54:20.898348 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:20.898427 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:20.898643 23272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:20.899116 23272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:20.900040 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23165
I20260808 22:54:20.900158 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:20.900350 22648 external_mini_cluster.cc:1428] Setting key e6544721c97d9ba71952caa86e069344
I20260808 22:54:20.900687 23272 catalog_manager.cc:1347] Generated new cluster ID: f9d1719b301644a2a902ac2f4d1e51fb
I20260808 22:54:20.900741 23272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:20.908882 23272 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:20.909032 23272 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:20.909057 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:20.900984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:57877 (local address 127.0.0.1:39547)
0808 22:54:20.901114 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:20.901122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:20.907591 (+ 6469us) server_negotiation.cc:407] Connection header received
0808 22:54:20.907735 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:20.907740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:20.907762 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:20.907890 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:20.908545 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:20.908548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:20.908548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:20.908551 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:20.908579 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:20.908597 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:20.908671 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:20.908795 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 22:54:20.908836 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
I20260808 22:54:20.910000 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:20.914939 23272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea65145557a642ea8a02453d2b791f41: Generated new TSK 0
I20260808 22:54:20.915125 23272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:20.916344 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:20.911634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:60805 (local address 127.0.0.1:39547)
0808 22:54:20.911665 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:20.911667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:20.915558 (+ 3891us) server_negotiation.cc:407] Connection header received
0808 22:54:20.915646 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:20.915647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:20.915650 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:20.915696 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:20.916158 (+ 462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:20.916159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:20.916159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:20.916162 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:20.916172 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:20.916176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:20.916212 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:20.916267 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 22:54:20.916276 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:54:20.917668 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23165
2026-08-08T22:54:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:20 dist-test-slave-x4qh krb5kdc[23309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:20 dist-test-slave-x4qh krb5kdc[23309](info): set up 2 sockets
Aug 08 22:54:20 dist-test-slave-x4qh krb5kdc[23309](info): commencing operation
krb5kdc: starting...
W20260808 22:54:22.078415 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:22 dist-test-slave-x4qh krb5kdc[23309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:22.181855 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:22.253224 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:22.253412 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:22.253448 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:22.254423 23328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:22.254495 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:22.254516 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:22.254542 23328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:22.254561 23328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:22.255842 23328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:22.256073 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:22.256279 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:22.260470 23346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:22.260484 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:22.260519 23328 server_base.cc:1034] running on GCE node
W20260808 22:54:22.260514 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:22.260898 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:22.261147 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:22.262320 23328 hybrid_clock.cc:648] HybridClock initialized: now 1786229662262310 us; error 28 us; skew 500 ppm
Aug 08 22:54:22 dist-test-slave-x4qh krb5kdc[23309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:22.264577 23328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:22.264945 23328 webserver.cc:460] Webserver started at http://127.0.0.1:38287/ using document root <none> and password file <none>
I20260808 22:54:22.265182 23328 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:22.265245 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:22.265367 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:22.266192 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "9d6372aa4c174c8983aadc2752c88369"
format_stamp: "Formatted at 2026-08-08 22:54:22 on dist-test-slave-x4qh"
server_key: "82e84ace760db8f47147266d49927cee"
server_key_iv: "bc762e444c1a05c48038a3a8af4b6d7c"
server_key_version: "encryptionkey@0"
I20260808 22:54:22.266445 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "9d6372aa4c174c8983aadc2752c88369"
format_stamp: "Formatted at 2026-08-08 22:54:22 on dist-test-slave-x4qh"
server_key: "82e84ace760db8f47147266d49927cee"
server_key_iv: "bc762e444c1a05c48038a3a8af4b6d7c"
server_key_version: "encryptionkey@0"
I20260808 22:54:22.267596 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:54:22.268239 23356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:22.268417 23328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:22.268477 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "9d6372aa4c174c8983aadc2752c88369"
format_stamp: "Formatted at 2026-08-08 22:54:22 on dist-test-slave-x4qh"
server_key: "82e84ace760db8f47147266d49927cee"
server_key_iv: "bc762e444c1a05c48038a3a8af4b6d7c"
server_key_version: "encryptionkey@0"
I20260808 22:54:22.268584 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:22.283583 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:22.285156 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:22.285301 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:22.289206 23328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32899
I20260808 22:54:22.289237 23429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32899 every 8 connection(s)
I20260808 22:54:22.289527 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:22.290273 23430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:22.292263 23430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369: Bootstrap starting.
I20260808 22:54:22.292783 23430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:22.293061 23430 log.cc:824] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:22.293668 23430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369: No bootstrap required, opened a new log
I20260808 22:54:22.295104 23430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6372aa4c174c8983aadc2752c88369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32899 } }
I20260808 22:54:22.295217 23430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:22.295257 23430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6372aa4c174c8983aadc2752c88369, State: Initialized, Role: FOLLOWER
I20260808 22:54:22.295442 23430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6372aa4c174c8983aadc2752c88369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32899 } }
I20260808 22:54:22.295504 23430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:22.295539 23430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:22.295584 23430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:22.295791 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23328
I20260808 22:54:22.295910 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:22.296073 22648 external_mini_cluster.cc:1428] Setting key a8c260e45c2792de5b6d0c4763b856c4
I20260808 22:54:22.296113 23430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6372aa4c174c8983aadc2752c88369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32899 } }
I20260808 22:54:22.296236 23430 leader_election.cc:302] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d6372aa4c174c8983aadc2752c88369; no voters:
I20260808 22:54:22.296384 23430 leader_election.cc:288] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:22.296419 23436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:22.296685 23436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6372aa4c174c8983aadc2752c88369, State: Running, Role: LEADER
I20260808 22:54:22.296761 23430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:22.296810 23436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6372aa4c174c8983aadc2752c88369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32899 } }
I20260808 22:54:22.297127 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d6372aa4c174c8983aadc2752c88369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6372aa4c174c8983aadc2752c88369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32899 } } }
I20260808 22:54:22.297255 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:22.297456 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6372aa4c174c8983aadc2752c88369. Latest consensus state: current_term: 1 leader_uuid: "9d6372aa4c174c8983aadc2752c88369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6372aa4c174c8983aadc2752c88369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32899 } } }
I20260808 22:54:22.297523 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:22.297899 23452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:22.298398 23452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:22.299867 23452 catalog_manager.cc:1347] Generated new cluster ID: fc43900cd3584b9ba8afae811accb2ac
I20260808 22:54:22.299908 23452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:22 dist-test-slave-x4qh krb5kdc[23309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 22:54:22.305842 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:22.296501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:47385 (local address 127.0.0.1:32899)
0808 22:54:22.296888 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:22.296896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:22.296927 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:54:22.297039 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:22.297046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:22.297097 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:22.298124 (+ 1027us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:22.298218 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:22.299303 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:22.300036 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:22.300464 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:22.300569 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:22.302683 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:22.302686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:22.302687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:22.302697 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:22.304461 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:22.304822 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:22.304827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:22.304827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:22.304922 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:22.305140 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:22.305141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:22.305142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:22.305350 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:22.305515 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:22.305595 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 22:54:22.305683 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":112,"threads_started":1}
I20260808 22:54:22.306654 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:22.317231 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:22.311643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:48957 (local address 127.0.0.1:32899)
0808 22:54:22.311680 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:22.311682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:22.311771 (+ 89us) server_negotiation.cc:407] Connection header received
0808 22:54:22.311810 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:22.311811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:22.311830 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:22.311876 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:22.312310 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:22.312705 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:22.313364 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:22.313565 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:22.313627 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:22.314752 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:22.314754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:22.314754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:22.314761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:22.316138 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:22.316494 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:22.316495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:22.316496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:22.316543 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:22.316772 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:22.316773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:22.316773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:22.316869 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:22.316965 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:22.317093 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 22:54:22.317145 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:54:22.320621 23452 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:22.321235 23452 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:22.327343 23452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d6372aa4c174c8983aadc2752c88369: Generated new TSK 0
I20260808 22:54:22.327541 23452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:22.333421 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23328
W20260808 22:54:22.344254 23307 connection.cc:535] client connection to 127.0.0.1:32899 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32899): Connection reset by peer (error 104)
2026-08-08T22:54:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:22 dist-test-slave-x4qh krb5kdc[23479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:22 dist-test-slave-x4qh krb5kdc[23479](info): set up 2 sockets
Aug 08 22:54:22 dist-test-slave-x4qh krb5kdc[23479](info): commencing operation
krb5kdc: starting...
W20260808 22:54:23.471918 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:23 dist-test-slave-x4qh krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:23.593830 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:23.667434 23502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:23.667658 23502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:23.667697 23502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:23.668776 23502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:23.668844 23502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:23.668867 23502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:23.668896 23502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:23.668920 23502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:23.670338 23502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:23.670598 23502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:23.670804 23502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:23.675135 23513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:23.675155 23517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:23.675288 23502 server_base.cc:1034] running on GCE node
W20260808 22:54:23.675159 23508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:23.675530 23502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:23.675788 23502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:23.676939 23502 hybrid_clock.cc:648] HybridClock initialized: now 1786229663676931 us; error 37 us; skew 500 ppm
Aug 08 22:54:23 dist-test-slave-x4qh krb5kdc[23479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:23.679190 23502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:23.679608 23502 webserver.cc:460] Webserver started at http://127.0.0.1:38113/ using document root <none> and password file <none>
I20260808 22:54:23.679801 23502 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:23.679862 23502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:23.679980 23502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:23.680868 23502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "cfd72ae34fdf4ce19622715a668511c4"
format_stamp: "Formatted at 2026-08-08 22:54:23 on dist-test-slave-x4qh"
server_key: "8c2cfacc266908c6cfc9eb02b8b9c18e"
server_key_iv: "30897bce2aab9b119962ac5347f69aef"
server_key_version: "encryptionkey@0"
I20260808 22:54:23.681182 23502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "cfd72ae34fdf4ce19622715a668511c4"
format_stamp: "Formatted at 2026-08-08 22:54:23 on dist-test-slave-x4qh"
server_key: "8c2cfacc266908c6cfc9eb02b8b9c18e"
server_key_iv: "30897bce2aab9b119962ac5347f69aef"
server_key_version: "encryptionkey@0"
I20260808 22:54:23.682312 23502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:54:23.682987 23527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:23.683132 23502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:23.683223 23502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "cfd72ae34fdf4ce19622715a668511c4"
format_stamp: "Formatted at 2026-08-08 22:54:23 on dist-test-slave-x4qh"
server_key: "8c2cfacc266908c6cfc9eb02b8b9c18e"
server_key_iv: "30897bce2aab9b119962ac5347f69aef"
server_key_version: "encryptionkey@0"
I20260808 22:54:23.683310 23502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:23.697144 23502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:23.698835 23502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:23.698987 23502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:23.702661 23502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39575
I20260808 22:54:23.702690 23606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39575 every 8 connection(s)
I20260808 22:54:23.702987 23502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:23.703706 23607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:23.705737 23607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4: Bootstrap starting.
I20260808 22:54:23.706283 23607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:23.706552 23607 log.cc:824] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:23.707150 23607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4: No bootstrap required, opened a new log
I20260808 22:54:23.708043 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23502
I20260808 22:54:23.708154 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:23.708326 22648 external_mini_cluster.cc:1428] Setting key a606d0e60c4322ece5e3c1289293eba4
I20260808 22:54:23.709208 23607 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd72ae34fdf4ce19622715a668511c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39575 } }
I20260808 22:54:23.709369 23607 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:23.709414 23607 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd72ae34fdf4ce19622715a668511c4, State: Initialized, Role: FOLLOWER
I20260808 22:54:23.709585 23607 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd72ae34fdf4ce19622715a668511c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39575 } }
I20260808 22:54:23.709668 23607 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:23.709702 23607 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:23.709760 23607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:23.710390 23607 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd72ae34fdf4ce19622715a668511c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39575 } }
I20260808 22:54:23.710502 23607 leader_election.cc:302] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd72ae34fdf4ce19622715a668511c4; no voters:
I20260808 22:54:23.710633 23607 leader_election.cc:288] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:23.710668 23615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:23.710807 23615 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [term 1 LEADER]: Becoming Leader. State: Replica: cfd72ae34fdf4ce19622715a668511c4, State: Running, Role: LEADER
I20260808 22:54:23.710841 23607 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:23.711524 23615 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd72ae34fdf4ce19622715a668511c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39575 } }
I20260808 22:54:23.712116 23616 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd72ae34fdf4ce19622715a668511c4. Latest consensus state: current_term: 1 leader_uuid: "cfd72ae34fdf4ce19622715a668511c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd72ae34fdf4ce19622715a668511c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39575 } } }
I20260808 22:54:23.712224 23616 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:23.712410 23618 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd72ae34fdf4ce19622715a668511c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd72ae34fdf4ce19622715a668511c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39575 } } }
I20260808 22:54:23.712544 23638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:23.713569 23618 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:23.713910 23638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:23.714447 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:23.708839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:45443 (local address 127.0.0.1:39575)
0808 22:54:23.709035 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:23.709044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:23.709075 (+ 31us) server_negotiation.cc:407] Connection header received
0808 22:54:23.709177 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:23.709194 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:23.709242 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:23.709408 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:23.709632 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:23.711135 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:23.712424 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:23.712878 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:23.712957 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:23.713668 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:23.713672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:23.713672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:23.713891 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:23.713925 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:23.713939 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:23.714017 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:23.714167 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 22:54:23.714193 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260808 22:54:23.715492 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:23.719678 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:23.716786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:58863 (local address 127.0.0.1:39575)
0808 22:54:23.716820 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:23.716822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:23.716884 (+ 62us) server_negotiation.cc:407] Connection header received
0808 22:54:23.716942 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:23.716943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:23.716952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:23.717006 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:23.717087 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:23.717750 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:23.718418 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:23.718765 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:23.718835 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:23.719458 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:23.719460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:23.719460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:23.719464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:23.719477 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:23.719482 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:23.719551 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:23.719598 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 22:54:23.719609 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:54:23.727722 23638 catalog_manager.cc:1347] Generated new cluster ID: 630fd6f5c6c440cc974882f9a910c3b3
I20260808 22:54:23.727778 23638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:23.743177 23638 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:23.743772 23638 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:23.751780 23638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd72ae34fdf4ce19622715a668511c4: Generated new TSK 0
I20260808 22:54:23.751953 23638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:23.761477 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23502
W20260808 22:54:23.773222 23476 connection.cc:535] client connection to 127.0.0.1:39575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39575): Connection reset by peer (error 104)
2026-08-08T22:54:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:23 dist-test-slave-x4qh krb5kdc[23654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:23 dist-test-slave-x4qh krb5kdc[23654](info): set up 2 sockets
Aug 08 22:54:23 dist-test-slave-x4qh krb5kdc[23654](info): commencing operation
krb5kdc: starting...
W20260808 22:54:24.915242 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:24 dist-test-slave-x4qh krb5kdc[23654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 22:54:25.023828 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:25.099489 23671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:25.099679 23671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:25.099716 23671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:25.100728 23671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:25.100793 23671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:25.100816 23671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:25.100842 23671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:25.100867 23671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:25.102195 23671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:25.102444 23671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:25.102650 23671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:25.107254 23677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:25.107302 23676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:25.107257 23671 server_base.cc:1034] running on GCE node
W20260808 22:54:25.107261 23679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:25.107697 23671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:25.107967 23671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:25.109104 23671 hybrid_clock.cc:648] HybridClock initialized: now 1786229665109094 us; error 26 us; skew 500 ppm
Aug 08 22:54:25 dist-test-slave-x4qh krb5kdc[23654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:25.111346 23671 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 22:54:25.111830 23671 webserver.cc:460] Webserver started at http://127.0.0.1:36049/ using document root <none> and password file <none>
I20260808 22:54:25.111995 23671 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:25.112032 23671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:25.112114 23671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:25.112980 23671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "9228290972db4111b731ecf9c87a9a82"
format_stamp: "Formatted at 2026-08-08 22:54:25 on dist-test-slave-x4qh"
server_key: "ec7cf90f36a0f124644eb8c2dbb17923"
server_key_iv: "c65763fda8959736705fce2cfb291f7a"
server_key_version: "encryptionkey@0"
I20260808 22:54:25.113253 23671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "9228290972db4111b731ecf9c87a9a82"
format_stamp: "Formatted at 2026-08-08 22:54:25 on dist-test-slave-x4qh"
server_key: "ec7cf90f36a0f124644eb8c2dbb17923"
server_key_iv: "c65763fda8959736705fce2cfb291f7a"
server_key_version: "encryptionkey@0"
I20260808 22:54:25.114356 23671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 22:54:25.114989 23688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:25.115140 23671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:25.115231 23671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "9228290972db4111b731ecf9c87a9a82"
format_stamp: "Formatted at 2026-08-08 22:54:25 on dist-test-slave-x4qh"
server_key: "ec7cf90f36a0f124644eb8c2dbb17923"
server_key_iv: "c65763fda8959736705fce2cfb291f7a"
server_key_version: "encryptionkey@0"
I20260808 22:54:25.115304 23671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:25.118424 23671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:25.120015 23671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:25.120146 23671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:25.123523 23671 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39705
I20260808 22:54:25.123550 23765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39705 every 8 connection(s)
I20260808 22:54:25.123813 23671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:25.124653 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:25.126457 23767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82: Bootstrap starting.
I20260808 22:54:25.126895 23767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:25.127094 23767 log.cc:824] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:25.127671 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23671
I20260808 22:54:25.127725 23767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82: No bootstrap required, opened a new log
I20260808 22:54:25.127863 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:25.128041 22648 external_mini_cluster.cc:1428] Setting key c656d3251c8adb0e4e6492e8f19b5309
I20260808 22:54:25.129559 23767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9228290972db4111b731ecf9c87a9a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 22:54:25.129709 23767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:25.129748 23767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9228290972db4111b731ecf9c87a9a82, State: Initialized, Role: FOLLOWER
I20260808 22:54:25.129910 23767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9228290972db4111b731ecf9c87a9a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 22:54:25.129997 23767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:25.130018 23767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:25.130080 23767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:25.130726 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:25.128507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:43105 (local address 127.0.0.1:39705)
0808 22:54:25.128682 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:25.128688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:25.128710 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:54:25.129547 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:25.129552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:25.129574 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:25.129668 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:25.130319 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:25.130322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:25.130323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:25.130326 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:25.130358 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:25.130381 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:25.130428 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:25.130515 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 22:54:25.130540 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260808 22:54:25.131089 23767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9228290972db4111b731ecf9c87a9a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 22:54:25.131175 23767 leader_election.cc:302] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9228290972db4111b731ecf9c87a9a82; no voters:
I20260808 22:54:25.131349 23767 leader_election.cc:288] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:25.131407 23773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:25.131536 23773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [term 1 LEADER]: Becoming Leader. State: Replica: 9228290972db4111b731ecf9c87a9a82, State: Running, Role: LEADER
I20260808 22:54:25.131551 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:25.131639 23767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:25.131662 23773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9228290972db4111b731ecf9c87a9a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } }
I20260808 22:54:25.131968 23773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9228290972db4111b731ecf9c87a9a82. Latest consensus state: current_term: 1 leader_uuid: "9228290972db4111b731ecf9c87a9a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9228290972db4111b731ecf9c87a9a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260808 22:54:25.131980 23774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9228290972db4111b731ecf9c87a9a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9228290972db4111b731ecf9c87a9a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39705 } } }
I20260808 22:54:25.132047 23773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:25.132097 23774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:25.132296 23786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:25.132771 23786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:25.134053 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:25.132939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.93:43083 (local address 127.0.0.1:39705)
0808 22:54:25.132969 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:25.132970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:25.133168 (+ 198us) server_negotiation.cc:407] Connection header received
0808 22:54:25.133262 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:25.133263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:25.133267 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:25.133302 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:25.133871 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:25.133872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:25.133872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:25.133876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:25.133886 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:25.133890 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:25.133929 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:25.133975 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 22:54:25.133987 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:54:25.134423 23786 catalog_manager.cc:1347] Generated new cluster ID: dc481cf80d5848a6b18ac7875ef6738a
I20260808 22:54:25.134487 23786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:25.138733 23786 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:25.138904 23786 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:25.149538 23786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9228290972db4111b731ecf9c87a9a82: Generated new TSK 0
I20260808 22:54:25.149684 23786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:25.151100 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23671
2026-08-08T22:54:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:25 dist-test-slave-x4qh krb5kdc[23818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:25 dist-test-slave-x4qh krb5kdc[23818](info): set up 2 sockets
Aug 08 22:54:25 dist-test-slave-x4qh krb5kdc[23818](info): commencing operation
krb5kdc: starting...
W20260808 22:54:26.323518 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:26 dist-test-slave-x4qh krb5kdc[23818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:26.434165 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:41265
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:41265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:26.508332 23833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:26.508541 23833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:26.508594 23833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:26.509820 23833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:26.509892 23833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:26.509917 23833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:26.509944 23833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:26.509967 23833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:26.511265 23833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:41265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:26.511576 23833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:26.511787 23833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:26.516364 23838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:26.516341 23839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:26.516454 23841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:26.516417 23833 server_base.cc:1034] running on GCE node
I20260808 22:54:26.516675 23833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:26.516891 23833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:26.518021 23833 hybrid_clock.cc:648] HybridClock initialized: now 1786229666517985 us; error 66 us; skew 500 ppm
Aug 08 22:54:26 dist-test-slave-x4qh krb5kdc[23818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229666, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:26.524937 23833 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:26.525398 23833 webserver.cc:460] Webserver started at http://127.22.30.62:40179/ using document root <none> and password file <none>
I20260808 22:54:26.525612 23833 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:26.525656 23833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:26.525833 23833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:26.526810 23833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "d5b657720c1748f6a1febe70eddbc21e"
format_stamp: "Formatted at 2026-08-08 22:54:26 on dist-test-slave-x4qh"
server_key: "16c15b487cd036ca6e18c804761bbc43"
server_key_iv: "84a07730457d548bc6091212bf11e51a"
server_key_version: "encryptionkey@0"
I20260808 22:54:26.527155 23833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "d5b657720c1748f6a1febe70eddbc21e"
format_stamp: "Formatted at 2026-08-08 22:54:26 on dist-test-slave-x4qh"
server_key: "16c15b487cd036ca6e18c804761bbc43"
server_key_iv: "84a07730457d548bc6091212bf11e51a"
server_key_version: "encryptionkey@0"
I20260808 22:54:26.528503 23833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:26.529198 23849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:26.529386 23833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:26.529474 23833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "d5b657720c1748f6a1febe70eddbc21e"
format_stamp: "Formatted at 2026-08-08 22:54:26 on dist-test-slave-x4qh"
server_key: "16c15b487cd036ca6e18c804761bbc43"
server_key_iv: "84a07730457d548bc6091212bf11e51a"
server_key_version: "encryptionkey@0"
I20260808 22:54:26.529548 23833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:26.537411 23833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:26.539469 23833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:26.539587 23833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:26.543660 23833 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:41265
I20260808 22:54:26.543723 23927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:41265 every 8 connection(s)
I20260808 22:54:26.544003 23833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:26.544808 23929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:26.546811 23929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e: Bootstrap starting.
I20260808 22:54:26.547313 23929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:26.547603 23929 log.cc:824] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:26.548319 23929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e: No bootstrap required, opened a new log
I20260808 22:54:26.548352 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23833
I20260808 22:54:26.548481 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:26.548659 22648 external_mini_cluster.cc:1428] Setting key 3ceb716256fa1ce04432e22e5c319669
I20260808 22:54:26.550676 23929 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b657720c1748f6a1febe70eddbc21e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 41265 } }
I20260808 22:54:26.550836 23929 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:26.550875 23929 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b657720c1748f6a1febe70eddbc21e, State: Initialized, Role: FOLLOWER
I20260808 22:54:26.551016 23929 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b657720c1748f6a1febe70eddbc21e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 41265 } }
I20260808 22:54:26.551074 23929 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:26.551115 23929 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:26.551156 23929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:26.551754 23929 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b657720c1748f6a1febe70eddbc21e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 41265 } }
I20260808 22:54:26.551862 23929 leader_election.cc:302] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5b657720c1748f6a1febe70eddbc21e; no voters:
I20260808 22:54:26.551995 23929 leader_election.cc:288] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:26.552066 23934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:26.552201 23929 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:26.552233 23934 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [term 1 LEADER]: Becoming Leader. State: Replica: d5b657720c1748f6a1febe70eddbc21e, State: Running, Role: LEADER
I20260808 22:54:26.552328 23934 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b657720c1748f6a1febe70eddbc21e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 41265 } }
I20260808 22:54:26.552647 23934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5b657720c1748f6a1febe70eddbc21e. Latest consensus state: current_term: 1 leader_uuid: "d5b657720c1748f6a1febe70eddbc21e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b657720c1748f6a1febe70eddbc21e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 41265 } } }
I20260808 22:54:26.552884 23934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:26.552810 23935 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5b657720c1748f6a1febe70eddbc21e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b657720c1748f6a1febe70eddbc21e" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 41265 } } }
I20260808 22:54:26.553272 23935 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:26.553292 23945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:26.553912 23945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:26.555685 23945 catalog_manager.cc:1347] Generated new cluster ID: 3c982a323877405ea0863a32ebc93258
I20260808 22:54:26.555738 23945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:26 dist-test-slave-x4qh krb5kdc[23818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:26.560009 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:26.549144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56535 (local address 127.22.30.62:41265)
0808 22:54:26.549327 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:26.549335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:26.549364 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:54:26.550635 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:26.550643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:26.550698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:26.550835 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:26.551580 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:26.552860 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:26.553949 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:26.554479 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:26.554573 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:26.556909 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:26.556912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:26.556913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:26.556923 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:26.558469 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:26.558861 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:26.558864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:26.558864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:26.558949 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:26.559191 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:26.559192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:26.559192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:26.559473 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:26.559644 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:26.559727 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 22:54:26.559823 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260808 22:54:26.560878 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:26.565397 23945 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:26.566032 23945 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:26.570291 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:26.563664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43857 (local address 127.22.30.62:41265)
0808 22:54:26.563697 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:26.563699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:26.563909 (+ 210us) server_negotiation.cc:407] Connection header received
0808 22:54:26.563972 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:26.563973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:26.563981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:26.564031 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:26.564503 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:26.564930 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:26.565926 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:26.566261 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:26.566364 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:26.567657 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:26.567658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:26.567658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:26.567665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:26.569152 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:26.569524 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:26.569526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:26.569526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:26.569582 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:26.569822 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:26.569823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:26.569823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:26.569941 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:26.570032 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:26.570142 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 22:54:26.570202 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":15}
I20260808 22:54:26.578843 23945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5b657720c1748f6a1febe70eddbc21e: Generated new TSK 0
I20260808 22:54:26.578998 23945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:26.581341 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23833
W20260808 22:54:26.594129 23816 connection.cc:535] client connection to 127.22.30.62:41265 recv error: Network error: failed to read from TLS socket (remote: 127.22.30.62:41265): Connection reset by peer (error 104)
2026-08-08T22:54:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:26 dist-test-slave-x4qh krb5kdc[23972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:26 dist-test-slave-x4qh krb5kdc[23972](info): set up 2 sockets
Aug 08 22:54:26 dist-test-slave-x4qh krb5kdc[23972](info): commencing operation
krb5kdc: starting...
W20260808 22:54:27.767230 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:27 dist-test-slave-x4qh krb5kdc[23972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:27.879163 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:44527
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:27.959023 23992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:27.959256 23992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:27.959295 23992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:27.960479 23992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:27.960579 23992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:27.960609 23992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:27.960646 23992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:27.960672 23992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:27.962108 23992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:44527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:27.962353 23992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:27.962603 23992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:27.967293 23998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:27.967316 23999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:27.967293 24001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:27.967350 23992 server_base.cc:1034] running on GCE node
I20260808 22:54:27.967828 23992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:27.968062 23992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:27.969209 23992 hybrid_clock.cc:648] HybridClock initialized: now 1786229667969198 us; error 32 us; skew 500 ppm
Aug 08 22:54:27 dist-test-slave-x4qh krb5kdc[23972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229667, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:27.971567 23992 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:27.972043 23992 webserver.cc:460] Webserver started at http://127.22.30.62:34483/ using document root <none> and password file <none>
I20260808 22:54:27.972265 23992 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:27.972313 23992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:27.972470 23992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:27.973508 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "df70a19a78d944e08dae21a17b081693"
format_stamp: "Formatted at 2026-08-08 22:54:27 on dist-test-slave-x4qh"
server_key: "b4e2aed29782153ef8f3eebf9f60a374"
server_key_iv: "adc89ec3daa7b616492d0eb965a6fb71"
server_key_version: "encryptionkey@0"
I20260808 22:54:27.973922 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "df70a19a78d944e08dae21a17b081693"
format_stamp: "Formatted at 2026-08-08 22:54:27 on dist-test-slave-x4qh"
server_key: "b4e2aed29782153ef8f3eebf9f60a374"
server_key_iv: "adc89ec3daa7b616492d0eb965a6fb71"
server_key_version: "encryptionkey@0"
I20260808 22:54:27.975289 23992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 22:54:27.976037 24013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:27.976194 23992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:27.976297 23992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "df70a19a78d944e08dae21a17b081693"
format_stamp: "Formatted at 2026-08-08 22:54:27 on dist-test-slave-x4qh"
server_key: "b4e2aed29782153ef8f3eebf9f60a374"
server_key_iv: "adc89ec3daa7b616492d0eb965a6fb71"
server_key_version: "encryptionkey@0"
I20260808 22:54:27.976378 23992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:27.988243 23992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:27.990049 23992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:27.990214 23992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:27.994489 23992 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:44527
I20260808 22:54:27.994514 24093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:44527 every 8 connection(s)
I20260808 22:54:27.994892 23992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:27.995702 24095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:27.997711 24095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693: Bootstrap starting.
I20260808 22:54:27.998234 24095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:27.998474 24095 log.cc:824] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:27.999125 24095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693: No bootstrap required, opened a new log
I20260808 22:54:28.000608 24095 raft_consensus.cc:348] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df70a19a78d944e08dae21a17b081693" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44527 } }
I20260808 22:54:28.000732 24095 raft_consensus.cc:374] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:28.000753 24095 raft_consensus.cc:729] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df70a19a78d944e08dae21a17b081693, State: Initialized, Role: FOLLOWER
I20260808 22:54:28.000928 24095 consensus_queue.cc:260] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df70a19a78d944e08dae21a17b081693" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44527 } }
I20260808 22:54:28.001015 24095 raft_consensus.cc:388] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:28.001121 24095 raft_consensus.cc:482] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:28.001181 24095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:28.002261 24095 raft_consensus.cc:504] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df70a19a78d944e08dae21a17b081693" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44527 } }
I20260808 22:54:28.002404 24095 leader_election.cc:302] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df70a19a78d944e08dae21a17b081693; no voters:
I20260808 22:54:28.002542 24095 leader_election.cc:288] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:28.002657 24098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:28.002766 24095 sys_catalog.cc:564] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:28.002830 24098 raft_consensus.cc:686] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [term 1 LEADER]: Becoming Leader. State: Replica: df70a19a78d944e08dae21a17b081693, State: Running, Role: LEADER
I20260808 22:54:28.002940 24098 consensus_queue.cc:237] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df70a19a78d944e08dae21a17b081693" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44527 } }
I20260808 22:54:28.003283 24099 sys_catalog.cc:455] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df70a19a78d944e08dae21a17b081693" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df70a19a78d944e08dae21a17b081693" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44527 } } }
I20260808 22:54:28.003284 24102 sys_catalog.cc:455] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df70a19a78d944e08dae21a17b081693. Latest consensus state: current_term: 1 leader_uuid: "df70a19a78d944e08dae21a17b081693" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df70a19a78d944e08dae21a17b081693" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 44527 } } }
I20260808 22:54:28.003410 24102 sys_catalog.cc:458] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:28.003408 24099 sys_catalog.cc:458] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:28.003644 24109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:28.004240 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 23992
I20260808 22:54:28.004359 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:28.004591 22648 external_mini_cluster.cc:1428] Setting key 9ec884f8bda83f14d2d9c495b54a895e
I20260808 22:54:28.004837 24109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:28.007026 24109 catalog_manager.cc:1347] Generated new cluster ID: 0cacfdf5d986403ba78259bd47a51924
I20260808 22:54:28.007078 24109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:28.010859 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:28.005259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48729 (local address 127.22.30.62:44527)
0808 22:54:28.005429 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:28.005437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:28.005629 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:54:28.005730 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:28.005735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:28.005775 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:28.005925 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:28.006169 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:28.007501 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:28.008600 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:28.009197 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:28.009276 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:28.010114 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:28.010118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:28.010118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:28.010317 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:28.010351 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:28.010364 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:28.010433 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:28.010637 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 22:54:28.010660 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260808 22:54:28.011693 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:28.016537 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:28.013007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41869 (local address 127.22.30.62:44527)
0808 22:54:28.013040 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:28.013041 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:28.013293 (+ 252us) server_negotiation.cc:407] Connection header received
0808 22:54:28.013344 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:28.013345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:28.013354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:28.013431 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:28.013567 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:28.014260 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:28.015278 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:28.015632 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:28.015687 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:28.016315 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:28.016316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:28.016316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:28.016320 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:28.016331 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:28.016335 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:28.016386 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:28.016462 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 22:54:28.016473 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260808 22:54:28.027771 24109 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:28.028440 24109 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:28.034869 24109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df70a19a78d944e08dae21a17b081693: Generated new TSK 0
I20260808 22:54:28.035032 24109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:28.042752 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 23992
W20260808 22:54:28.056900 23970 connection.cc:535] client connection to 127.22.30.62:44527 recv error: Network error: failed to read from TLS socket (remote: 127.22.30.62:44527): Connection reset by peer (error 104)
2026-08-08T22:54:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:28 dist-test-slave-x4qh krb5kdc[24134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:28 dist-test-slave-x4qh krb5kdc[24134](info): set up 2 sockets
Aug 08 22:54:28 dist-test-slave-x4qh krb5kdc[24134](info): commencing operation
krb5kdc: starting...
W20260808 22:54:29.214016 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:29 dist-test-slave-x4qh krb5kdc[24134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:29.322110 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33013
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:33013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:29.397791 24159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:29.397964 24159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:29.398002 24159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:29.399120 24159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:29.399160 24159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:29.399174 24159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:29.399186 24159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:29.399235 24159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:29.400558 24159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:33013
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:33013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:29.400740 24159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:29.400951 24159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:29.405650 24167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:29.405731 24159 server_base.cc:1034] running on GCE node
W20260808 22:54:29.405680 24166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:29.405680 24170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:29.406096 24159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:29.406375 24159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:29.407533 24159 hybrid_clock.cc:648] HybridClock initialized: now 1786229669407518 us; error 34 us; skew 500 ppm
Aug 08 22:54:29 dist-test-slave-x4qh krb5kdc[24134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:29.409901 24159 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:29.410295 24159 webserver.cc:460] Webserver started at http://127.22.30.62:34577/ using document root <none> and password file <none>
I20260808 22:54:29.410527 24159 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:29.410593 24159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:29.410722 24159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:29.411787 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "4ef1c03d79f8402aa0229b5b53dd07f4"
format_stamp: "Formatted at 2026-08-08 22:54:29 on dist-test-slave-x4qh"
server_key: "1860bc08e0bba1ebb9ac9db7e11b1fda"
server_key_iv: "5b261986f512a9a2a866dc465b579eb8"
server_key_version: "encryptionkey@0"
I20260808 22:54:29.412137 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "4ef1c03d79f8402aa0229b5b53dd07f4"
format_stamp: "Formatted at 2026-08-08 22:54:29 on dist-test-slave-x4qh"
server_key: "1860bc08e0bba1ebb9ac9db7e11b1fda"
server_key_iv: "5b261986f512a9a2a866dc465b579eb8"
server_key_version: "encryptionkey@0"
I20260808 22:54:29.413424 24159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:29.414079 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:29.414250 24159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:29.414348 24159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "4ef1c03d79f8402aa0229b5b53dd07f4"
format_stamp: "Formatted at 2026-08-08 22:54:29 on dist-test-slave-x4qh"
server_key: "1860bc08e0bba1ebb9ac9db7e11b1fda"
server_key_iv: "5b261986f512a9a2a866dc465b579eb8"
server_key_version: "encryptionkey@0"
I20260808 22:54:29.414427 24159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:29.417444 24159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:29.419073 24159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:29.419230 24159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:29.422802 24159 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:33013
I20260808 22:54:29.422850 24256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:33013 every 8 connection(s)
I20260808 22:54:29.423143 24159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:29.424103 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:29.425971 24258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4: Bootstrap starting.
I20260808 22:54:29.426223 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24159
I20260808 22:54:29.426362 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:29.426548 22648 external_mini_cluster.cc:1428] Setting key 324a9622ca918bc19386b79dcb3135f0
I20260808 22:54:29.426647 24258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:29.426949 24258 log.cc:824] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:29.427685 24258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4: No bootstrap required, opened a new log
I20260808 22:54:29.429076 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:29.427020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35863 (local address 127.22.30.62:33013)
0808 22:54:29.427214 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:29.427222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:29.427251 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:54:29.427865 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:29.427873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:29.427902 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:29.428059 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:29.428659 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:29.428663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:29.428663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:29.428667 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:29.428708 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:29.428722 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:29.428823 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:29.428854 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 22:54:29.428890 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260808 22:54:29.430119 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:29.430114 24258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33013 } }
I20260808 22:54:29.430256 24258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:29.430300 24258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef1c03d79f8402aa0229b5b53dd07f4, State: Initialized, Role: FOLLOWER
I20260808 22:54:29.430397 24258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33013 } }
I20260808 22:54:29.430465 24258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:29.430490 24258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:29.430533 24258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:29.431178 24258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33013 } }
I20260808 22:54:29.431301 24258 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef1c03d79f8402aa0229b5b53dd07f4; no voters:
I20260808 22:54:29.431480 24258 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:29.431501 24270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:29.431680 24270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef1c03d79f8402aa0229b5b53dd07f4, State: Running, Role: LEADER
I20260808 22:54:29.431792 24258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:29.431823 24270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33013 } }
I20260808 22:54:29.432174 24273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef1c03d79f8402aa0229b5b53dd07f4. Latest consensus state: current_term: 1 leader_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33013 } } }
I20260808 22:54:29.432281 24273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:29.432159 24270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef1c03d79f8402aa0229b5b53dd07f4" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 33013 } } }
I20260808 22:54:29.432660 24270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:29.432819 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:29.431594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39927 (local address 127.22.30.62:33013)
0808 22:54:29.431632 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:29.431633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:29.431786 (+ 153us) server_negotiation.cc:407] Connection header received
0808 22:54:29.431879 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:29.431880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:29.431884 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:29.431949 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:29.432584 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:29.432585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:29.432586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:29.432589 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:29.432601 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:29.432605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:29.432633 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:29.432740 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 22:54:29.432752 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:29.432837 24282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:29.433537 24282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:29.434796 24282 catalog_manager.cc:1347] Generated new cluster ID: 4c8218ab1bc548b2a37e323716236e85
I20260808 22:54:29.434845 24282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:29.443066 24282 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:29.443217 24282 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:29.446419 24282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef1c03d79f8402aa0229b5b53dd07f4: Generated new TSK 0
I20260808 22:54:29.446553 24282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:29.450626 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24159
2026-08-08T22:54:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:29 dist-test-slave-x4qh krb5kdc[24301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:29 dist-test-slave-x4qh krb5kdc[24301](info): set up 2 sockets
Aug 08 22:54:29 dist-test-slave-x4qh krb5kdc[24301](info): commencing operation
krb5kdc: starting...
W20260808 22:54:30.619622 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:30 dist-test-slave-x4qh krb5kdc[24301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:30.728046 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34461
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:34213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:34461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:30.802074 24320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:30.802238 24320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:30.802258 24320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:30.803344 24320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:30.803400 24320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:30.803416 24320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:30.803462 24320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:30.803486 24320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:30.804966 24320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:34213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:34461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:30.805197 24320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:30.805409 24320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:30.809986 24329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:30.809988 24330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:30.810029 24335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:30.810061 24320 server_base.cc:1034] running on GCE node
I20260808 22:54:30.810415 24320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:30.810737 24320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:30.811866 24320 hybrid_clock.cc:648] HybridClock initialized: now 1786229670811869 us; error 38 us; skew 500 ppm
Aug 08 22:54:30 dist-test-slave-x4qh krb5kdc[24301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229670, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:30.814090 24320 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:30.814533 24320 webserver.cc:460] Webserver started at http://127.22.30.62:39679/ using document root <none> and password file <none>
I20260808 22:54:30.814750 24320 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:30.814812 24320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:30.814935 24320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:30.815986 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "6dbb6a77523549078a1b99b7e3bed472"
format_stamp: "Formatted at 2026-08-08 22:54:30 on dist-test-slave-x4qh"
server_key: "965b87551f7161c67d1a865ec0c5c0b7"
server_key_iv: "3163d617205ad7fd8f5424ea1a98feda"
server_key_version: "encryptionkey@0"
I20260808 22:54:30.816346 24320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "6dbb6a77523549078a1b99b7e3bed472"
format_stamp: "Formatted at 2026-08-08 22:54:30 on dist-test-slave-x4qh"
server_key: "965b87551f7161c67d1a865ec0c5c0b7"
server_key_iv: "3163d617205ad7fd8f5424ea1a98feda"
server_key_version: "encryptionkey@0"
I20260808 22:54:30.817523 24320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:30.818205 24343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:30.818372 24320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:30.818457 24320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "6dbb6a77523549078a1b99b7e3bed472"
format_stamp: "Formatted at 2026-08-08 22:54:30 on dist-test-slave-x4qh"
server_key: "965b87551f7161c67d1a865ec0c5c0b7"
server_key_iv: "3163d617205ad7fd8f5424ea1a98feda"
server_key_version: "encryptionkey@0"
I20260808 22:54:30.818531 24320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:30.845397 24320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:30.847146 24320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:30.847330 24320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:30.851207 24320 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:34461
I20260808 22:54:30.851240 24420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:34461 every 8 connection(s)
I20260808 22:54:30.851734 24320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:30.852250 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24320
I20260808 22:54:30.852392 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:30.852433 24421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:30.852597 22648 external_mini_cluster.cc:1428] Setting key bc71ad7f355b4bec5730ac74eaefea9d
I20260808 22:54:30.855571 24421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472: Bootstrap starting.
I20260808 22:54:30.856134 24421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:30.856454 24421 log.cc:824] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:30.857149 24421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472: No bootstrap required, opened a new log
Aug 08 22:54:30 dist-test-slave-x4qh krb5kdc[24301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:30.859426 24421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbb6a77523549078a1b99b7e3bed472" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34461 } }
I20260808 22:54:30.859588 24421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:30.859611 24421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dbb6a77523549078a1b99b7e3bed472, State: Initialized, Role: FOLLOWER
I20260808 22:54:30.859733 24421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbb6a77523549078a1b99b7e3bed472" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34461 } }
I20260808 22:54:30.859812 24421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:30.859843 24421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:30.859891 24421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:30.860458 24421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbb6a77523549078a1b99b7e3bed472" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34461 } }
I20260808 22:54:30.860630 24421 leader_election.cc:302] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dbb6a77523549078a1b99b7e3bed472; no voters:
I20260808 22:54:30.860917 24421 leader_election.cc:288] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:30.860954 24429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:30.861083 24429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [term 1 LEADER]: Becoming Leader. State: Replica: 6dbb6a77523549078a1b99b7e3bed472, State: Running, Role: LEADER
I20260808 22:54:30.861192 24421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:30.861227 24429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbb6a77523549078a1b99b7e3bed472" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34461 } }
I20260808 22:54:30.861596 24429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dbb6a77523549078a1b99b7e3bed472. Latest consensus state: current_term: 1 leader_uuid: "6dbb6a77523549078a1b99b7e3bed472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbb6a77523549078a1b99b7e3bed472" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34461 } } }
I20260808 22:54:30.861714 24430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dbb6a77523549078a1b99b7e3bed472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dbb6a77523549078a1b99b7e3bed472" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34461 } } }
I20260808 22:54:30.861771 24429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:30.861799 24430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:30.861985 24441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:30.862583 24441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:30.863235 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:30.853096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38475 (local address 127.22.30.62:34461)
0808 22:54:30.853305 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:30.853314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:30.853346 (+ 32us) server_negotiation.cc:407] Connection header received
0808 22:54:30.853794 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:30.853803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:30.853857 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:30.853993 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:30.854513 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:30.855719 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:30.856442 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:30.856950 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:30.857043 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:30.859512 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:30.859516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:30.859516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:30.859524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:30.861269 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:30.861617 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:30.861619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:30.861619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:30.861733 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:30.861976 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:30.861976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:30.861977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:30.862256 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:30.862860 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:30.862940 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 22:54:30.863076 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260808 22:54:30.864158 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:30.867813 24441 catalog_manager.cc:1347] Generated new cluster ID: 00a215ed406b48c6b64efa2513761ea3
I20260808 22:54:30.867869 24441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:30.870914 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:30.865410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58591 (local address 127.22.30.62:34461)
0808 22:54:30.865444 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:30.865446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:30.865534 (+ 88us) server_negotiation.cc:407] Connection header received
0808 22:54:30.865583 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:30.865583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:30.865591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:30.865638 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:30.866072 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:30.866463 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:30.867092 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:30.867340 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:30.867440 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:30.868518 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:30.868519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:30.868520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:30.868527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:30.869877 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:30.870207 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:30.870208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:30.870208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:30.870264 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:30.870463 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:30.870464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:30.870464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:30.870576 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:30.870663 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:30.870760 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 22:54:30.870832 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260808 22:54:30.883082 24441 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:30.883622 24441 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:30.886677 24441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dbb6a77523549078a1b99b7e3bed472: Generated new TSK 0
I20260808 22:54:30.886824 24441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:30.888988 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24320
2026-08-08T22:54:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:30 dist-test-slave-x4qh krb5kdc[24460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:30 dist-test-slave-x4qh krb5kdc[24460](info): set up 2 sockets
Aug 08 22:54:30 dist-test-slave-x4qh krb5kdc[24460](info): commencing operation
krb5kdc: starting...
W20260808 22:54:32.044483 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:32 dist-test-slave-x4qh krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:32.153713 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39279
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:39279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:32.229286 24479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:32.229507 24479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:32.229544 24479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:32.230611 24479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:32.230677 24479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:32.230703 24479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:32.230726 24479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:32.230746 24479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:32.232187 24479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:39279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:39279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:32.232438 24479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:32.232652 24479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:32.237241 24487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:32.237237 24490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:32.237250 24488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:32.237404 24479 server_base.cc:1034] running on GCE node
I20260808 22:54:32.237663 24479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:32.237934 24479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:32.239099 24479 hybrid_clock.cc:648] HybridClock initialized: now 1786229672239083 us; error 27 us; skew 500 ppm
Aug 08 22:54:32 dist-test-slave-x4qh krb5kdc[24460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229672, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:32.246152 24479 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:32.246562 24479 webserver.cc:460] Webserver started at http://127.22.30.62:34651/ using document root <none> and password file <none>
I20260808 22:54:32.246805 24479 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:32.246876 24479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:32.247025 24479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:32.248113 24479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "f7cb18e8fd2546afa809cedfc564241b"
format_stamp: "Formatted at 2026-08-08 22:54:32 on dist-test-slave-x4qh"
server_key: "05c35e3de90da70a9822fc62235d65ce"
server_key_iv: "86b9a4adc65c72838c757785f0f6a442"
server_key_version: "encryptionkey@0"
I20260808 22:54:32.248466 24479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "f7cb18e8fd2546afa809cedfc564241b"
format_stamp: "Formatted at 2026-08-08 22:54:32 on dist-test-slave-x4qh"
server_key: "05c35e3de90da70a9822fc62235d65ce"
server_key_iv: "86b9a4adc65c72838c757785f0f6a442"
server_key_version: "encryptionkey@0"
I20260808 22:54:32.249758 24479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 22:54:32.250473 24499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:32.250648 24479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:32.250746 24479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "f7cb18e8fd2546afa809cedfc564241b"
format_stamp: "Formatted at 2026-08-08 22:54:32 on dist-test-slave-x4qh"
server_key: "05c35e3de90da70a9822fc62235d65ce"
server_key_iv: "86b9a4adc65c72838c757785f0f6a442"
server_key_version: "encryptionkey@0"
I20260808 22:54:32.250834 24479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:32.268402 24479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:32.270042 24479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:32.270193 24479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:32.274032 24479 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:39279
I20260808 22:54:32.274065 24579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:39279 every 8 connection(s)
I20260808 22:54:32.274372 24479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:32.275197 24580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:32.277313 24580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b: Bootstrap starting.
I20260808 22:54:32.277778 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24479
I20260808 22:54:32.277907 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:32.277915 24580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:32.278124 22648 external_mini_cluster.cc:1428] Setting key 2fe97417c3278d20b208d64809774fe4
I20260808 22:54:32.278241 24580 log.cc:824] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:32.279078 24580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b: No bootstrap required, opened a new log
I20260808 22:54:32.281278 24580 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb18e8fd2546afa809cedfc564241b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39279 } }
I20260808 22:54:32.281430 24580 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:32.281459 24580 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7cb18e8fd2546afa809cedfc564241b, State: Initialized, Role: FOLLOWER
I20260808 22:54:32.281566 24580 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb18e8fd2546afa809cedfc564241b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39279 } }
I20260808 22:54:32.281633 24580 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:32.281667 24580 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:32.281725 24580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:32.282276 24580 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb18e8fd2546afa809cedfc564241b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39279 } }
I20260808 22:54:32.282385 24580 leader_election.cc:302] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7cb18e8fd2546afa809cedfc564241b; no voters:
I20260808 22:54:32.282557 24580 leader_election.cc:288] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:32.282599 24585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:32.282719 24585 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [term 1 LEADER]: Becoming Leader. State: Replica: f7cb18e8fd2546afa809cedfc564241b, State: Running, Role: LEADER
I20260808 22:54:32.282829 24580 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:32.282866 24585 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb18e8fd2546afa809cedfc564241b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39279 } }
I20260808 22:54:32.283617 24586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7cb18e8fd2546afa809cedfc564241b. Latest consensus state: current_term: 1 leader_uuid: "f7cb18e8fd2546afa809cedfc564241b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb18e8fd2546afa809cedfc564241b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39279 } } }
I20260808 22:54:32.283695 24586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:32.283857 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:32.278644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42081 (local address 127.22.30.62:39279)
0808 22:54:32.278812 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:32.278820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:32.278857 (+ 37us) server_negotiation.cc:407] Connection header received
0808 22:54:32.279304 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:32.279311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:32.279354 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:32.279559 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:32.279795 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:32.280637 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:32.281751 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:32.282200 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:32.282290 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:32.282988 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:32.282992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:32.282992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:32.283164 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:32.283192 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:32.283202 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:32.283248 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:32.283551 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 22:54:32.283626 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260808 22:54:32.284065 24587 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7cb18e8fd2546afa809cedfc564241b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7cb18e8fd2546afa809cedfc564241b" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 39279 } } }
I20260808 22:54:32.284132 24587 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:32.284382 24598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:32.284952 24598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:32.284971 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:32.286484 24598 catalog_manager.cc:1347] Generated new cluster ID: 9d1c4c0b03614641be32b1f917f4dbf6
I20260808 22:54:32.286540 24598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:32.289117 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:32.286321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55691 (local address 127.22.30.62:39279)
0808 22:54:32.286359 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:32.286360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:32.286473 (+ 113us) server_negotiation.cc:407] Connection header received
0808 22:54:32.286583 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:32.286584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:32.286594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:32.286645 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:32.286750 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:32.287196 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:32.287958 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:32.288158 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:32.288258 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:32.288890 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:32.288891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:32.288891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:32.288895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:32.288907 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:32.288911 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:32.288958 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:32.289013 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 22:54:32.289031 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 22:54:32.300491 24598 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:32.300992 24598 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:32.303960 24598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7cb18e8fd2546afa809cedfc564241b: Generated new TSK 0
I20260808 22:54:32.304111 24598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:32.314011 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24479
2026-08-08T22:54:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:32 dist-test-slave-x4qh krb5kdc[24621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:32 dist-test-slave-x4qh krb5kdc[24621](info): set up 2 sockets
Aug 08 22:54:32 dist-test-slave-x4qh krb5kdc[24621](info): commencing operation
krb5kdc: starting...
W20260808 22:54:33.480093 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:33 dist-test-slave-x4qh krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:33.595098 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34239
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:34239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:33.672446 24639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:33.672655 24639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:33.672693 24639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:33.673790 24639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:33.673853 24639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:33.673877 24639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:33.673903 24639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:33.673928 24639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:33.675263 24639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:34239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:33.675562 24639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:33.675782 24639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:33.680279 24647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:33.680322 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:33.680406 24639 server_base.cc:1034] running on GCE node
W20260808 22:54:33.680522 24651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:33.680745 24639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:33.681031 24639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:33.682173 24639 hybrid_clock.cc:648] HybridClock initialized: now 1786229673682161 us; error 33 us; skew 500 ppm
Aug 08 22:54:33 dist-test-slave-x4qh krb5kdc[24621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229673, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:33.684659 24639 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:33.684999 24639 webserver.cc:460] Webserver started at http://127.22.30.62:40799/ using document root <none> and password file <none>
I20260808 22:54:33.685207 24639 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:33.685248 24639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:33.685335 24639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:33.686270 24639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "1a55fc5764ce4172a9ea4e0030c83978"
format_stamp: "Formatted at 2026-08-08 22:54:33 on dist-test-slave-x4qh"
server_key: "d45c73eeb07699378b2b42c3cc8389c4"
server_key_iv: "b758a717924c21082d81be17af28c9f1"
server_key_version: "encryptionkey@0"
I20260808 22:54:33.686560 24639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "1a55fc5764ce4172a9ea4e0030c83978"
format_stamp: "Formatted at 2026-08-08 22:54:33 on dist-test-slave-x4qh"
server_key: "d45c73eeb07699378b2b42c3cc8389c4"
server_key_iv: "b758a717924c21082d81be17af28c9f1"
server_key_version: "encryptionkey@0"
I20260808 22:54:33.687855 24639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:33.688501 24663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:33.688668 24639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:33.688756 24639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "1a55fc5764ce4172a9ea4e0030c83978"
format_stamp: "Formatted at 2026-08-08 22:54:33 on dist-test-slave-x4qh"
server_key: "d45c73eeb07699378b2b42c3cc8389c4"
server_key_iv: "b758a717924c21082d81be17af28c9f1"
server_key_version: "encryptionkey@0"
I20260808 22:54:33.688841 24639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:33.692412 24639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:33.694412 24639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:33.694553 24639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:33.698223 24639 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:34239
I20260808 22:54:33.698254 24742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:34239 every 8 connection(s)
I20260808 22:54:33.698575 24639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:33.699249 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24639
I20260808 22:54:33.699417 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:33.699643 22648 external_mini_cluster.cc:1428] Setting key fe7659c49a5cb31da10168e9e6a9a3ee
I20260808 22:54:33.699781 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:33.701792 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:33.700129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38263 (local address 127.22.30.62:34239)
0808 22:54:33.700306 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:33.700313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:33.700342 (+ 29us) server_negotiation.cc:407] Connection header received
0808 22:54:33.700704 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:33.700709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:33.700729 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:33.700857 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:33.701403 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:33.701406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:33.701406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:33.701410 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:33.701446 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:33.701461 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:33.701525 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:33.701598 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 22:54:33.701633 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260808 22:54:33.702673 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:33.702756 24744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978: Bootstrap starting.
I20260808 22:54:33.703316 24744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:33.703666 24744 log.cc:824] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:33.704442 24744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978: No bootstrap required, opened a new log
I20260808 22:54:33.705027 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:33.703979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48369 (local address 127.22.30.62:34239)
0808 22:54:33.704039 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:33.704040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:33.704106 (+ 66us) server_negotiation.cc:407] Connection header received
0808 22:54:33.704176 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:33.704177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:33.704180 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:33.704221 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:33.704827 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:33.704828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:33.704828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 22:54:33.704832 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:33.704847 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 22:54:33.704851 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:33.704919 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:33.704959 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 22:54:33.704970 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 22:54:33.706322 24744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a55fc5764ce4172a9ea4e0030c83978" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34239 } }
I20260808 22:54:33.706431 24744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 22:54:33.706451 24744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a55fc5764ce4172a9ea4e0030c83978, State: Initialized, Role: FOLLOWER
I20260808 22:54:33.706517 24744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a55fc5764ce4172a9ea4e0030c83978" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34239 } }
I20260808 22:54:33.706568 24744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 22:54:33.706584 24744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 22:54:33.706610 24744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:33.707024 24744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a55fc5764ce4172a9ea4e0030c83978" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34239 } }
I20260808 22:54:33.707098 24744 leader_election.cc:302] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a55fc5764ce4172a9ea4e0030c83978; no voters:
I20260808 22:54:33.707203 24744 leader_election.cc:288] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 22:54:33.707275 24757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:33.707445 24744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:33.707474 24757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [term 1 LEADER]: Becoming Leader. State: Replica: 1a55fc5764ce4172a9ea4e0030c83978, State: Running, Role: LEADER
I20260808 22:54:33.707721 24757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a55fc5764ce4172a9ea4e0030c83978" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34239 } }
I20260808 22:54:33.708097 24760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a55fc5764ce4172a9ea4e0030c83978. Latest consensus state: current_term: 1 leader_uuid: "1a55fc5764ce4172a9ea4e0030c83978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a55fc5764ce4172a9ea4e0030c83978" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34239 } } }
I20260808 22:54:33.708105 24758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a55fc5764ce4172a9ea4e0030c83978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a55fc5764ce4172a9ea4e0030c83978" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 34239 } } }
I20260808 22:54:33.708242 24760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:33.708276 24758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978 [sys.catalog]: This master's current role is: LEADER
W20260808 22:54:33.709343 24780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:33.709399 24780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:33.709494 24769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:33.709678 24769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:33.710963 24769 catalog_manager.cc:1347] Generated new cluster ID: ee962c2a0c384758ad2b02af4cf65eeb
I20260808 22:54:33.711026 24769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:33.724400 24769 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:33.724545 24769 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:33.726949 24769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a55fc5764ce4172a9ea4e0030c83978: Generated new TSK 0
I20260808 22:54:33.727108 24769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 22:54:33.736232 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24639
2026-08-08T22:54:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1419 ms)
[----------] 18 tests from AuthTokenIssuingTest (25596 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:33 dist-test-slave-x4qh krb5kdc[24786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:33 dist-test-slave-x4qh krb5kdc[24786](info): set up 2 sockets
Aug 08 22:54:33 dist-test-slave-x4qh krb5kdc[24786](info): commencing operation
krb5kdc: starting...
W20260808 22:54:34.898640 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:34 dist-test-slave-x4qh krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:35.005815 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37381
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37381,127.22.30.61:42519,127.22.30.60:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:35.081681 24804 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:54:35.081878 24804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:35.081907 24804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:35.081962 24804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:35.082960 24804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:35.083020 24804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:35.083045 24804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:35.083073 24804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:35.083098 24804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:35.084501 24804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37381,127.22.30.61:42519,127.22.30.60:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:35.084744 24804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:35.084950 24804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:35.089545 24810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:35.089623 24811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:35.089547 24815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:35.091522 24804 server_base.cc:1034] running on GCE node
I20260808 22:54:35.091773 24804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:35.092020 24804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:35.093179 24804 hybrid_clock.cc:648] HybridClock initialized: now 1786229675093179 us; error 38 us; skew 500 ppm
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:35.095358 24804 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:35.095827 24804 webserver.cc:460] Webserver started at http://127.22.30.62:44749/ using document root <none> and password file <none>
I20260808 22:54:35.096048 24804 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:35.096112 24804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:35.096235 24804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:35.097179 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "dc239b47f43748a8a20a57c8d8f04774"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "28f32ae3711e1809df0e489f70e8fb48"
server_key_iv: "339e82e0ad2d14d48bce12c2ee089898"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.097543 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "dc239b47f43748a8a20a57c8d8f04774"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "28f32ae3711e1809df0e489f70e8fb48"
server_key_iv: "339e82e0ad2d14d48bce12c2ee089898"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.098737 24804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:35.099670 24825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:35.099864 24804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 22:54:35.099953 24804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "dc239b47f43748a8a20a57c8d8f04774"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "28f32ae3711e1809df0e489f70e8fb48"
server_key_iv: "339e82e0ad2d14d48bce12c2ee089898"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.100028 24804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:35.109674 24804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:35.111248 24804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:35.111435 24804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:35.115149 24804 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:37381
I20260808 22:54:35.115175 24914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:37381 every 8 connection(s)
I20260808 22:54:35.115509 24804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:35.116339 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:35.117620 24915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.119949 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24804
I20260808 22:54:35.120065 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:35.120253 22648 external_mini_cluster.cc:1428] Setting key 02d900c95b343223f52462b55ac2d162
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:35.126991 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.118164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37381 (local address 127.0.0.1:58050)
0808 22:54:35.118320 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.118355 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.118766 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.119010 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.119012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.119225 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.119425 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.119431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.120544 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.120546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.121515 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.121518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.121964 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.121965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.122024 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.122482 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.122489 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.123722 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.125541 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.125547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.125549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.125743 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.125999 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.125999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.126000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.126087 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.126433 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.126438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.126495 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.126599 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.126826 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":98,"thread_start_us":68,"threads_started":1}
I20260808 22:54:35.126996 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.118216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.22.30.62:37381)
0808 22:54:35.118389 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.118392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.118419 (+ 27us) server_negotiation.cc:407] Connection header received
0808 22:54:35.118895 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.118896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.118911 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.118981 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.119517 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.120463 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.121592 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.121865 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.121981 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.123826 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.123833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.123833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.123840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.125403 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.125844 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.125846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.125847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.125924 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.126153 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.126154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.126154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.126349 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.126437 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.126634 (+ 197us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.126761 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260808 22:54:35.127722 24915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.128000 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.127911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47838)
0808 22:54:35.127946 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.127982 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260808 22:54:35.128134 24826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.22.30.61@KRBTEST.COM; defaulting to no policy
I20260808 22:54:35.128536 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.128459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47854)
0808 22:54:35.128482 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.128521 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260808 22:54:35.128661 24915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:42519: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab.
Entry for principal kudu/127.22.30.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab.
I20260808 22:54:35.143590 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:42519
--webserver_interface=127.22.30.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab
--principal=kudu/127.22.30.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37381,127.22.30.61:42519,127.22.30.60:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:54:35.177968 24915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } attempt: 1
I20260808 22:54:35.178321 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.178204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47864)
0808 22:54:35.178254 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.178294 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 22:54:35.178653 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.178564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47872)
0808 22:54:35.178606 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.178633 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 22:54:35.178756 24915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:42519: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
W20260808 22:54:35.223213 24922 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:54:35.223469 24922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:35.223505 24922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:35.223541 24922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:35.224612 24922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:35.224663 24922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:35.224676 24922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:35.224718 24922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:35.224745 24922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:35.226055 24922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37381,127.22.30.61:42519,127.22.30.60:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab
--principal=kudu/127.22.30.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.30.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:35.226275 24922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:35.226464 24922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:35.230993 24932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:35.231024 24930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:35.230993 24935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:35.231019 24922 server_base.cc:1034] running on GCE node
I20260808 22:54:35.231472 24922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:35.231721 24922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:35.232901 24922 hybrid_clock.cc:648] HybridClock initialized: now 1786229675232891 us; error 38 us; skew 500 ppm
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:35.235016 24922 init.cc:377] Logged in from keytab as kudu/127.22.30.61@KRBTEST.COM (short username kudu)
I20260808 22:54:35.235477 24922 webserver.cc:460] Webserver started at http://127.22.30.61:38533/ using document root <none> and password file <none>
I20260808 22:54:35.235682 24922 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:35.235719 24922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:35.235870 24922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:35.236764 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data/instance:
uuid: "b1473b24c32b4924999eb9e1cb79ec72"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "6a41a0ee1cc9f1fded20fbbdcbff13a0"
server_key_iv: "7de32f23cac8751f87462bd1ea1c4f53"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.237082 24922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal/instance:
uuid: "b1473b24c32b4924999eb9e1cb79ec72"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "6a41a0ee1cc9f1fded20fbbdcbff13a0"
server_key_iv: "7de32f23cac8751f87462bd1ea1c4f53"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.238294 24922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:35.238987 24942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:35.239287 24922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:54:35.239373 24922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal
uuid: "b1473b24c32b4924999eb9e1cb79ec72"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "6a41a0ee1cc9f1fded20fbbdcbff13a0"
server_key_iv: "7de32f23cac8751f87462bd1ea1c4f53"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.239459 24922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:35.246914 24915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } attempt: 2
I20260808 22:54:35.247264 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.247139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47888)
0808 22:54:35.247195 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.247232 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 22:54:35.247629 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.247556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47890)
0808 22:54:35.247584 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.247611 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260808 22:54:35.247751 24915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:42519: Network error: Client connection negotiation failed: client connection to 127.22.30.61:42519: connect: Connection refused (error 111)
I20260808 22:54:35.251636 24922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:35.253135 24922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:35.253288 24922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:35.257228 24922 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.61:42519
I20260808 22:54:35.257256 25013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.61:42519 every 8 connection(s)
I20260808 22:54:35.257553 24922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
I20260808 22:54:35.257745 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 24922
I20260808 22:54:35.257853 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-1/wal/instance
I20260808 22:54:35.258035 22648 external_mini_cluster.cc:1428] Setting key 406b8ac436e3dbd7c70ad197e1d5398a
I20260808 22:54:35.258482 25014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:35.259924 25014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.30.60@KRBTEST.COM; defaulting to no policy
I20260808 22:54:35.268209 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.260533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.22.30.62:37381)
0808 22:54:35.260593 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.260596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.260750 (+ 154us) server_negotiation.cc:407] Connection header received
0808 22:54:35.261135 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.261137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.261150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.261212 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.261679 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.262527 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.263508 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.263834 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.263900 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.265743 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.265745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.265745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.265752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.266942 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.267324 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.267325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.267325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.267406 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.267634 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.267635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.267635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.267759 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.267824 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.268026 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.268108 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.60@KRBTEST.COM" created.
I20260808 22:54:35.268301 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.260497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37381 (local address 127.0.0.1:58060)
0808 22:54:35.260636 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.260671 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.261054 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.261254 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.261256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.261444 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.261594 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.261597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.262631 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.262631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.263359 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.263362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.263931 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.263931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.264026 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.264489 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.264496 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.265655 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.267010 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.267017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.267018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.267262 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.267476 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.267477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.267477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.267565 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.267824 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.267829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.267923 (+ 94us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.267998 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.268095 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260808 22:54:35.268827 25014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:54:35.275473 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.269050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47896)
0808 22:54:35.269098 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.269108 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.269142 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.269433 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.269434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.269569 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.269607 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.269608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.270400 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.270401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.270871 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.270873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.271324 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.271325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.271405 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.271816 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.271827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.273024 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.274530 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.274531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.274531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.274676 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.274863 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.274864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.274864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.274913 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.275177 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.275177 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.275209 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.275270 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.275350 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 22:54:35.275537 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.269135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.22.30.61:42519)
0808 22:54:35.269292 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.269296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.269311 (+ 15us) server_negotiation.cc:407] Connection header received
0808 22:54:35.269328 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.269329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.269351 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.269432 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.269693 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.270316 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.270951 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.271231 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.271323 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.273102 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.273109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.273110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.273115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.274442 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.274748 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.274750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.274751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.274802 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.274988 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.274988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.274988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.275103 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.275193 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.275301 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.275428 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260808 22:54:35.276157 25014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.276480 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.276343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50830)
0808 22:54:35.276400 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.276437 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 22:54:35.276593 24946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
I20260808 22:54:35.276993 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.276910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50846)
0808 22:54:35.276955 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.276975 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 22:54:35.277084 25014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:32885: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab.
Entry for principal kudu/127.22.30.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab.
I20260808 22:54:35.282022 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:32885
--webserver_interface=127.22.30.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab
--principal=kudu/127.22.30.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:37381,127.22.30.61:42519,127.22.30.60:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:54:35.326268 25014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } attempt: 1
I20260808 22:54:35.326656 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.326538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50860)
0808 22:54:35.326600 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.326634 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 22:54:35.326987 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.326893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50876)
0808 22:54:35.326940 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.326964 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 22:54:35.327076 25014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:32885: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
I20260808 22:54:35.338955 24915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } attempt: 3
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:54:35.345731 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.339279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.22.30.61:42519)
0808 22:54:35.339325 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.339327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.339338 (+ 11us) server_negotiation.cc:407] Connection header received
0808 22:54:35.339398 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.339399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.339409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.339463 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.339792 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.340225 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.340888 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.341098 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.341167 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.343288 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.343289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.343289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.343294 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.344618 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.344996 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.344997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.344997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.345043 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.345248 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.345248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.345248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.345346 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.345391 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.345569 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.345669 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 22:54:35.345727 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.339215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47904)
0808 22:54:35.339272 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.339282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.339331 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.339478 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.339479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.339632 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.339691 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.339692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.340309 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.340311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.340801 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.340802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.341166 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.341167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.341248 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.341718 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.341722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.343199 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.344697 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.344698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.344699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.344920 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.345106 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.345107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.345107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.345184 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.345408 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.345409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.345459 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.345533 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.345633 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 22:54:35.346161 24915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.346484 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.346368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50882)
0808 22:54:35.346423 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.346454 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:54:35.346812 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.346724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50886)
0808 22:54:35.346770 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.346794 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 22:54:35.346926 24915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:32885: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
W20260808 22:54:35.360316 25024 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:54:35.360483 25024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:35.360500 25024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:35.360518 25024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:35.361479 25024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:35.361510 25024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:35.361522 25024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:35.361533 25024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:35.361544 25024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:35.362886 25024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:37381,127.22.30.61:42519,127.22.30.60:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab
--principal=kudu/127.22.30.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.30.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:35.363051 25024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:35.363194 25024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:35.367950 25034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:35.367935 25032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:35.367928 25036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:35.368119 25024 server_base.cc:1034] running on GCE node
I20260808 22:54:35.368373 25024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:35.368642 25024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:35.369789 25024 hybrid_clock.cc:648] HybridClock initialized: now 1786229675369777 us; error 27 us; skew 500 ppm
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:35.372105 25024 init.cc:377] Logged in from keytab as kudu/127.22.30.60@KRBTEST.COM (short username kudu)
I20260808 22:54:35.372506 25024 webserver.cc:460] Webserver started at http://127.22.30.60:46441/ using document root <none> and password file <none>
I20260808 22:54:35.372727 25024 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:35.372803 25024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:35.372956 25024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:35.373962 25024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data/instance:
uuid: "a4ef255069e54dd48e327af69c8de433"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "2d76308bf578273fdbc258d907b6c673"
server_key_iv: "5436b64a59a752a265c0badcff7beb61"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.374337 25024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal/instance:
uuid: "a4ef255069e54dd48e327af69c8de433"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "2d76308bf578273fdbc258d907b6c673"
server_key_iv: "5436b64a59a752a265c0badcff7beb61"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.375721 25024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 22:54:35.376446 25044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:35.376600 25024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:35.376691 25024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal
uuid: "a4ef255069e54dd48e327af69c8de433"
format_stamp: "Formatted at 2026-08-08 22:54:35 on dist-test-slave-x4qh"
server_key: "2d76308bf578273fdbc258d907b6c673"
server_key_iv: "5436b64a59a752a265c0badcff7beb61"
server_key_version: "encryptionkey@0"
I20260808 22:54:35.376761 25024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:35.378084 24915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } attempt: 1
I20260808 22:54:35.378423 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.378306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50888)
0808 22:54:35.378366 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.378397 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 22:54:35.378741 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.378687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50894)
0808 22:54:35.378708 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.378727 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260808 22:54:35.378844 24915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:32885: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
I20260808 22:54:35.394768 25024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:35.395210 25014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } attempt: 2
I20260808 22:54:35.395586 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.395453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50902)
0808 22:54:35.395520 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.395553 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 22:54:35.395948 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.395849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50916)
0808 22:54:35.395898 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.395927 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 22:54:35.396050 25014 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:32885: Network error: Client connection negotiation failed: client connection to 127.22.30.60:32885: connect: Connection refused (error 111)
I20260808 22:54:35.396875 25024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:35.397018 25024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:35.400848 25024 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.60:32885
I20260808 22:54:35.400885 25115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.60:32885 every 8 connection(s)
I20260808 22:54:35.401208 25024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
I20260808 22:54:35.402195 25116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:35.403795 25116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.406468 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25024
I20260808 22:54:35.406589 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786229410580610-22648-0/minicluster-data/master-2/wal/instance
I20260808 22:54:35.406788 22648 external_mini_cluster.cc:1428] Setting key 075c1aa1df520d15f1e872f32d9cec59
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:54:35.412562 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.404423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.22.30.62:37381)
0808 22:54:35.404482 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.404484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.404679 (+ 195us) server_negotiation.cc:407] Connection header received
0808 22:54:35.405146 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.405147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.405158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.405207 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.405752 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.406197 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.407461 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.407760 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.407831 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.409791 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.409792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.409792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.409799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.411162 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.411631 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.411632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.411633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.411685 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.411956 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.411957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.411957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.412070 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.412137 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.412399 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.412480 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 22:54:35.412724 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.404399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:37381 (local address 127.0.0.1:58076)
0808 22:54:35.404575 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.404606 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.405039 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.405276 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.405279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.405491 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.405669 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.405673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.406319 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.406320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.407339 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.407342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.407867 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.407868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.407963 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.408421 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.408428 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.409694 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.411261 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.411275 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.411277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.411550 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.411767 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.411768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.411768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.411887 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.412142 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.412148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.412279 (+ 131us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.412376 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.412508 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260808 22:54:35.413544 25116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.415230 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.407168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.22.30.60:32885)
0808 22:54:35.407323 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.407327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.407399 (+ 72us) server_negotiation.cc:407] Connection header received
0808 22:54:35.407499 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.407503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.407523 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.407589 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.408253 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.408933 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.409739 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.410024 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.410102 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.412211 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.412219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.412220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.412226 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.414262 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.414562 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.414565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.414565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.414619 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.414772 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.414772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.414778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.414933 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.415026 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.415124 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.415177 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:35.420323 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.413823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.22.30.61:42519)
0808 22:54:35.413861 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.413863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.413870 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:35.413893 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.413894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.413903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.413966 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.414207 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.414797 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.415584 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.415893 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.415956 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.417702 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.417704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.417704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.417711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.419189 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.419578 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.419580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.419580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.419631 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.419828 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.419829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.419829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.419913 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.419957 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.420148 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.420234 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:54:35.420598 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.413754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:42519 (local address 127.0.0.1:47918)
0808 22:54:35.413795 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.413804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.413835 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.413967 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.413967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.414116 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.414141 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.414142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.414860 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.414861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.415513 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.415515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.415963 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.415964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.416021 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.416394 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.416398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.417615 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.419271 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.419272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.419272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.419496 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.419696 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.419697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.419697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.419761 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.419973 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.419974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.420017 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.420407 (+ 390us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.420503 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260808 22:54:35.421391 25116 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:35.423055 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.415987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.22.30.62:37381)
0808 22:54:35.416023 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.416024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.416046 (+ 22us) server_negotiation.cc:407] Connection header received
0808 22:54:35.416063 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.416064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.416071 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.416111 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.416525 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.417149 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.417852 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.418141 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.418206 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.420460 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.420462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.420462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.420468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.422078 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.422437 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.422438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.422438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.422483 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.422652 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.422653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.422653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.422734 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.422878 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.422912 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.422977 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:54:35.427964 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.421603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50938)
0808 22:54:35.421663 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.421673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.421714 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.421832 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.421833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.421970 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.422005 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.422006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.422825 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.422826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.423587 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.423590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.423945 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.423945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.423988 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.424333 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.424337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.425336 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.426869 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.426870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.426870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.427059 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.427233 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.427233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.427234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.427296 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.427727 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.427728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.427760 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.427808 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.427914 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 22:54:35.428341 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.421670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.22.30.60:32885)
0808 22:54:35.421713 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.421715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.421723 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:35.421780 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.421781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.421788 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.421826 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.422072 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.422737 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.423677 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.423869 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.423943 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.425409 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.425410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.425411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.425417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.426811 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.427133 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.427134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.427134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.427176 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.427552 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.427553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.427553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.427641 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.427733 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.427835 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.428225 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":280,"server-negotiator.queue_time_us":28}
I20260808 22:54:35.430099 25116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: Bootstrap starting.
I20260808 22:54:35.430603 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.423552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.22.30.61:42519)
0808 22:54:35.423589 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.423591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.423600 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:35.423621 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.423621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.423630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.423679 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.424076 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.424588 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.425229 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.425487 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.425559 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.427702 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.427703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.427703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.427709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.429499 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.429815 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.429816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.429816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.429872 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.430034 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.430034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.430035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.430150 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.430283 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.430342 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.430501 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:35.431030 25116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:35.431332 25116 log.cc:824] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:35.432092 25116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: No bootstrap required, opened a new log
I20260808 22:54:35.432086 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:35.433411 25116 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.433555 25116 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ef255069e54dd48e327af69c8de433, State: Initialized, Role: FOLLOWER
I20260808 22:54:35.433645 25116 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.433032 24874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dc239b47f43748a8a20a57c8d8f04774"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58092
I20260808 22:54:35.433943 25116 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:35.433897 25123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.434002 25123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:54:35.435886 25144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:35.435983 25144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:35.435956 25024 master_runner.cc:186] Error getting master registration for 127.22.30.62:37381: OK, instance_id { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" instance_seqno: 1786229675114103 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:54:35.438630 25024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:54:35.453982 24915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } attempt: 2
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:54:35.461436 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.454318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.22.30.60:32885)
0808 22:54:35.454372 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.454374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.454381 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:35.454402 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.454403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.454411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.454473 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.454800 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.455529 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.456535 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.456831 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.456893 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.458792 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.458794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.458795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.458802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.460269 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.460634 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.460635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.460636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.460688 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.460897 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.460898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.460899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.461030 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.461103 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.461273 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.461341 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:54:35.461438 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.454234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50940)
0808 22:54:35.454288 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.454297 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.454333 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.454501 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.454502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.454675 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.454719 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.454721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.455609 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.455611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.456440 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.456443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.456907 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.456908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.456969 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.457441 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.457446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.458683 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.460371 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.460372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.460372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.460547 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.460763 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.460764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.460764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.460823 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.461109 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.461110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.461154 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.461233 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.461338 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 22:54:35.462788 24915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774: Bootstrap starting.
I20260808 22:54:35.463439 24915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:35.463740 24915 log.cc:824] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:35.464469 24915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774: No bootstrap required, opened a new log
I20260808 22:54:35.465461 24915 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.465585 24915 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc239b47f43748a8a20a57c8d8f04774, State: Initialized, Role: FOLLOWER
I20260808 22:54:35.465674 24915 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.465943 25148 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.465994 24915 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:35.466017 25148 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:35.466115 24874 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 22:54:35.466224 24874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:35.467039 24874 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.467728 24874 leader_election.cc:288] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [CANDIDATE]: Term 1 election: Requested vote from peers b1473b24c32b4924999eb9e1cb79ec72 (127.22.30.61:42519), a4ef255069e54dd48e327af69c8de433 (127.22.30.60:32885)
W20260808 22:54:35.467760 25166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:35.467921 25166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:35.467944 25076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc239b47f43748a8a20a57c8d8f04774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a4ef255069e54dd48e327af69c8de433"
I20260808 22:54:35.468050 25076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:35.468972 25076 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc239b47f43748a8a20a57c8d8f04774 in term 1.
I20260808 22:54:35.469291 24830 leader_election.cc:302] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ef255069e54dd48e327af69c8de433, dc239b47f43748a8a20a57c8d8f04774; no voters:
I20260808 22:54:35.469411 25148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:35.469538 25148 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [term 1 LEADER]: Becoming Leader. State: Replica: dc239b47f43748a8a20a57c8d8f04774, State: Running, Role: LEADER
I20260808 22:54:35.469626 25148 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.470042 25148 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc239b47f43748a8a20a57c8d8f04774. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.470125 25148 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:35.470477 25181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:35.470716 25181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:35.469575 24804 master_runner.cc:186] Error getting master registration for 127.22.30.61:42519: OK, instance_id { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" instance_seqno: 1786229675256143 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:54:35.471762 24804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:54:35.483652 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.469861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.22.30.60:32885)
0808 22:54:35.469902 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.469904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.472083 (+ 2179us) server_negotiation.cc:407] Connection header received
0808 22:54:35.472176 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.472177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.472187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.472217 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.472992 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.473711 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.475521 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.475791 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.475852 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.479740 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.479742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.479742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.479749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.481854 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.482356 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.482357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.482357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.482459 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.482765 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.482766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.482766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.482949 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.483072 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.483352 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.483491 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 22:54:35.483812 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.469405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.22.30.62:37381)
0808 22:54:35.469438 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.469440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.470776 (+ 1336us) server_negotiation.cc:407] Connection header received
0808 22:54:35.470941 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.470942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.470961 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.470987 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.473019 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.473709 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.474696 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.474961 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.475020 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.476932 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.476933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.476933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.476941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.478650 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.482638 (+ 3988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.482639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.482640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.482687 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.483210 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.483211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.483211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.483336 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.483437 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.483617 (+ 180us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.483686 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 22:54:35.484233 25076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [term 1 FOLLOWER]: Refusing update from remote peer dc239b47f43748a8a20a57c8d8f04774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:35.484529 25148 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:35.485809 25123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc239b47f43748a8a20a57c8d8f04774. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.485889 25123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:35.487129 25014 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } attempt: 3
I20260808 22:54:35.487294 25184 mvcc.cc:205] Tried to move back new op lower bound from 7316396750780817408 to 7316396750724403200. Current Snapshot: MvccSnapshot[applied={T|T < 7316396750780817408}]
I20260808 22:54:35.487726 25148 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.487808 25148 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:35.488426 25123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.488497 25123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:35.488817 25185 mvcc.cc:205] Tried to move back new op lower bound from 7316396750780817408 to 7316396750724403200. Current Snapshot: MvccSnapshot[applied={T|T < 7316396750780817408}]
I20260808 22:54:35.491677 25181 catalog_manager.cc:1347] Generated new cluster ID: e37e66a383ca4879b599d845fb2b3b3b
I20260808 22:54:35.491724 25181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 22:54:35 dist-test-slave-x4qh krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229675, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:54:35.502131 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.470352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.22.30.61:42519)
0808 22:54:35.470392 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.470393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.477022 (+ 6629us) server_negotiation.cc:407] Connection header received
0808 22:54:35.477123 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.477124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.477135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.477411 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.479756 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.480661 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.485226 (+ 4565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.485527 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.485570 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.491126 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.491127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.491128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.491134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.493430 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.501235 (+ 7805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.501236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.501236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.501284 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.501497 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.501498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.501498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.501630 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.501707 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.501933 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.502018 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:35.502468 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.487352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:32885 (local address 127.0.0.1:50956)
0808 22:54:35.487456 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:35.487466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:35.487507 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:35.487805 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:35.487806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:35.487962 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:35.487999 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.488000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.488860 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.488861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.489649 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:35.489652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.490149 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.490150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:35.490201 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.490622 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:35.490627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:35.493719 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:35.501180 (+ 7461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.501181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.501181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.501421 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.501668 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:35.501668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:35.501669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:35.501739 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:35.502130 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:35.502131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:35.502185 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 22:54:35.502247 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 22:54:35.502345 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260808 22:54:35.503060 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:35.487577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.22.30.60:32885)
0808 22:54:35.487622 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.487624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.487631 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:35.487650 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.487651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.487660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.487716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.488100 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.488746 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.489753 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:35.490054 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:35.490106 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:35.499445 (+ 9339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:35.499446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:35.499446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:35.499453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:35.501101 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.501515 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.501516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.501516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.501582 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:35.501852 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:35.501853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:35.501853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:35.502000 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:35.502079 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:35.502925 (+ 846us) server_negotiation.cc:299] Negotiation successful
0808 22:54:35.502988 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 22:54:35.504341 25014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: Bootstrap starting.
I20260808 22:54:35.505055 25014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:35.505373 25014 log.cc:824] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:35.506337 25014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: No bootstrap required, opened a new log
I20260808 22:54:35.508131 25014 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.508239 25014 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1473b24c32b4924999eb9e1cb79ec72, State: Initialized, Role: FOLLOWER
I20260808 22:54:35.508304 25014 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } }
I20260808 22:54:35.508586 25014 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:35.509840 24976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc239b47f43748a8a20a57c8d8f04774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b1473b24c32b4924999eb9e1cb79ec72"
I20260808 22:54:35.509943 24976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:35.510679 24976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc239b47f43748a8a20a57c8d8f04774 in term 1.
I20260808 22:54:35.510855 24975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [term 1 FOLLOWER]: Refusing update from remote peer dc239b47f43748a8a20a57c8d8f04774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:35.511104 25188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:35.515177 25192 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.515872 25192 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:35.515949 25192 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc239b47f43748a8a20a57c8d8f04774. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.515995 25192 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:35.516043 25192 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.516124 25192 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 22:54:35.516850 25207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:35.516901 25207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:35.523520 25212 mvcc.cc:205] Tried to move back new op lower bound from 7316396750780817408 to 7316396750724403200. Current Snapshot: MvccSnapshot[applied={T|T < 7316396750780817408}]
I20260808 22:54:35.523993 25181 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:35.524685 25181 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:35.531663 25188 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc239b47f43748a8a20a57c8d8f04774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc239b47f43748a8a20a57c8d8f04774" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 37381 } } peers { permanent_uuid: "b1473b24c32b4924999eb9e1cb79ec72" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 42519 } } peers { permanent_uuid: "a4ef255069e54dd48e327af69c8de433" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 32885 } } }
I20260808 22:54:35.531761 25188 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774 [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:35.543424 25181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc239b47f43748a8a20a57c8d8f04774: Generated new TSK 0
I20260808 22:54:35.543589 25181 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:54:35.568851 24918 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.567616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.22.30.62:37381)
0808 22:54:35.567656 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.567658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.568210 (+ 552us) server_negotiation.cc:407] Connection header received
0808 22:54:35.568313 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.568314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.568324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.568354 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.568781 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58100: BlockingRecv error: recv got EOF from 127.0.0.1:58100 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 22:54:35.569787 25121 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.567942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.22.30.60:32885)
0808 22:54:35.567980 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.567982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.569078 (+ 1096us) server_negotiation.cc:407] Connection header received
0808 22:54:35.569397 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.569398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.569407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.569436 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.569717 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50964: BlockingRecv error: recv got EOF from 127.0.0.1:50964 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 22:54:35.570581 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.569257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.22.30.61:42519)
0808 22:54:35.569293 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.569295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.570022 (+ 727us) server_negotiation.cc:407] Connection header received
0808 22:54:35.570170 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.570170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.570179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.570207 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.570528 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47940: BlockingRecv error: recv got EOF from 127.0.0.1:47940 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 22:54:35.572686 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.572018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.22.30.61:42519)
0808 22:54:35.572048 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.572050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.572199 (+ 149us) server_negotiation.cc:407] Connection header received
0808 22:54:35.572297 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.572298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.572314 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.572341 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.572648 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47950: BlockingRecv error: recv got EOF from 127.0.0.1:47950 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260808 22:54:35.575652 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.574916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.22.30.61:42519)
0808 22:54:35.574945 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.574947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.575139 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:54:35.575226 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.575227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.575235 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.575262 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.575613 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47960: BlockingRecv error: recv got EOF from 127.0.0.1:47960 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260808 22:54:35.579455 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.578695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.22.30.61:42519)
0808 22:54:35.578724 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.578726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.578922 (+ 196us) server_negotiation.cc:407] Connection header received
0808 22:54:35.579073 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.579074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.579083 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.579110 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.579418 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47972: BlockingRecv error: recv got EOF from 127.0.0.1:47972 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260808 22:54:35.585163 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.584484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.22.30.61:42519)
0808 22:54:35.584513 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.584515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.584696 (+ 181us) server_negotiation.cc:407] Connection header received
0808 22:54:35.584812 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.584813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.584822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.584849 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.585124 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47976: BlockingRecv error: recv got EOF from 127.0.0.1:47976 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 22:54:35.594823 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.594146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.22.30.61:42519)
0808 22:54:35.594174 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.594175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.594367 (+ 192us) server_negotiation.cc:407] Connection header received
0808 22:54:35.594451 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.594452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.594460 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.594484 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.594789 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47986: BlockingRecv error: recv got EOF from 127.0.0.1:47986 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260808 22:54:35.613332 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.612337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.22.30.61:42519)
0808 22:54:35.612382 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.612384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.612645 (+ 261us) server_negotiation.cc:407] Connection header received
0808 22:54:35.612751 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.612752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.612763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.612808 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.613272 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47992: BlockingRecv error: recv got EOF from 127.0.0.1:47992 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 22:54:35.647213 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.646380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.22.30.61:42519)
0808 22:54:35.646424 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.646426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.646590 (+ 164us) server_negotiation.cc:407] Connection header received
0808 22:54:35.646634 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.646635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.646648 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.646726 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.647147 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47994: BlockingRecv error: recv got EOF from 127.0.0.1:47994 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 22:54:35.713088 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.712345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48000 (local address 127.22.30.61:42519)
0808 22:54:35.712425 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.712427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.712532 (+ 105us) server_negotiation.cc:407] Connection header received
0808 22:54:35.712615 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.712616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.712628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.712695 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.713033 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48000: BlockingRecv error: recv got EOF from 127.0.0.1:48000 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260808 22:54:35.842965 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:35.842227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.22.30.61:42519)
0808 22:54:35.842295 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:35.842297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:35.842408 (+ 111us) server_negotiation.cc:407] Connection header received
0808 22:54:35.842483 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:35.842484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:35.842495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:35.842565 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:35.842852 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48002: BlockingRecv error: recv got EOF from 127.0.0.1:48002 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260808 22:54:36.100819 25019 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:36.100181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.22.30.61:42519)
0808 22:54:36.100247 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:36.100249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:36.100262 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:54:36.100347 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:36.100348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:36.100358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:36.100406 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:36.100724 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48012: BlockingRecv error: recv got EOF from 127.0.0.1:48012 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 22:54:36.436421 25144 catalog_manager.cc:1259] Loaded cluster ID: e37e66a383ca4879b599d845fb2b3b3b
I20260808 22:54:36.436527 25144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: loading cluster ID for follower catalog manager: success
I20260808 22:54:36.437299 25144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: acquiring CA information for follower catalog manager: success
I20260808 22:54:36.437613 25144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a4ef255069e54dd48e327af69c8de433: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:54:36.517320 25207 catalog_manager.cc:1259] Loaded cluster ID: e37e66a383ca4879b599d845fb2b3b3b
I20260808 22:54:36.517385 25207 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: loading cluster ID for follower catalog manager: success
I20260808 22:54:36.518185 25207 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: acquiring CA information for follower catalog manager: success
I20260808 22:54:36.518500 25207 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b1473b24c32b4924999eb9e1cb79ec72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:54:36.616334 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:36.613881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.22.30.61:42519)
0808 22:54:36.614017 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:36.614019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:36.614079 (+ 60us) server_negotiation.cc:407] Connection header received
0808 22:54:36.614128 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:36.614128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:36.614177 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:36.614249 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:36.614406 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.614860 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.615622 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.615810 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.615862 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:36.615996 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:36.616098 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:36.616155 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:36.616190 (+ 35us) server_negotiation.cc:299] Negotiation successful
0808 22:54:36.616207 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260808 22:54:36.620306 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:36.618140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.22.30.60:32885)
0808 22:54:36.618238 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:36.618239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:36.618246 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:36.618286 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:36.618287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:36.618318 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:36.618375 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:36.618468 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.618913 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.619613 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.619847 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.619895 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:36.620007 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:36.620091 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:36.620140 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:36.620173 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 22:54:36.620189 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":58,"threads_started":1}
I20260808 22:54:36.624641 25333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:36.621886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.22.30.62:37381)
0808 22:54:36.622007 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:36.622010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:36.622024 (+ 14us) server_negotiation.cc:407] Connection header received
0808 22:54:36.622146 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:36.622147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:36.622196 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:36.622274 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:36.622444 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.622914 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.623843 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.624020 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.624101 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:36.624300 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:36.624406 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:36.624461 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:36.624514 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 22:54:36.624539 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260808 22:54:36.624755 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:36.622046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.22.30.60:32885)
0808 22:54:36.622089 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:36.622091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:36.622236 (+ 145us) server_negotiation.cc:407] Connection header received
0808 22:54:36.622305 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:36.622306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:36.622315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:36.622354 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:36.622484 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.623112 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.624096 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.624284 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.624379 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:36.624532 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:36.624560 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:36.624586 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:36.624678 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 22:54:36.624696 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:36.625227 25315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:36.621836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.22.30.61:42519)
0808 22:54:36.621872 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:36.621874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:36.622043 (+ 169us) server_negotiation.cc:407] Connection header received
0808 22:54:36.622138 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:36.622139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:36.622148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:36.622193 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:36.622336 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.623040 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.624076 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:36.624341 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:36.624401 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:36.624671 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:36.624703 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:36.624734 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:36.625103 (+ 369us) server_negotiation.cc:299] Negotiation successful
0808 22:54:36.625127 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 22:54:36.626519 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24804
I20260808 22:54:36.645157 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 24922
I20260808 22:54:36.655908 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25024
2026-08-08T22:54:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2918 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 22:54:36 dist-test-slave-x4qh krb5kdc[25340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 22:54:36 dist-test-slave-x4qh krb5kdc[25340](info): set up 2 sockets
Aug 08 22:54:36 dist-test-slave-x4qh krb5kdc[25340](info): commencing operation
krb5kdc: starting...
W20260808 22:54:37.810189 22648 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/test-user.keytab.
Aug 08 22:54:37 dist-test-slave-x4qh krb5kdc[25340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T22:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T22:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.30.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
Entry for principal kudu/127.22.30.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab.
I20260808 22:54:37.919662 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42035
--webserver_interface=127.22.30.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:42035,127.22.30.61:37863,127.22.30.60:37997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 22:54:37.996198 25359 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:54:37.996393 25359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:37.996409 25359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:37.996426 25359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:37.997392 25359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:37.997423 25359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:37.997437 25359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:37.997449 25359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:37.997462 25359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:37.998724 25359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:42035,127.22.30.61:37863,127.22.30.60:37997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.62.keytab
--principal=kudu/127.22.30.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.62:42035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.30.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:37.998919 25359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:37.999061 25359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 22:54:38.003425 25372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:38.003430 25367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:38.003468 25359 server_base.cc:1034] running on GCE node
W20260808 22:54:38.003458 25368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:38.003902 25359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:38.004182 25359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:38.005321 25359 hybrid_clock.cc:648] HybridClock initialized: now 1786229678005301 us; error 34 us; skew 500 ppm
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:38.013298 25359 init.cc:377] Logged in from keytab as kudu/127.22.30.62@KRBTEST.COM (short username kudu)
I20260808 22:54:38.013765 25359 webserver.cc:460] Webserver started at http://127.22.30.62:41551/ using document root <none> and password file <none>
I20260808 22:54:38.013983 25359 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:38.014052 25359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:38.014200 25359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:38.015141 25359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data/instance:
uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "c3beb4ea82c7c439ee98c331243fc9b4"
server_key_iv: "5ec6b22bcc1d916da4e3442d29144c13"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.015525 25359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal/instance:
uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "c3beb4ea82c7c439ee98c331243fc9b4"
server_key_iv: "5ec6b22bcc1d916da4e3442d29144c13"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.016810 25359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 22:54:38.017453 25381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:38.017606 25359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 22:54:38.017695 25359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal
uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "c3beb4ea82c7c439ee98c331243fc9b4"
server_key_iv: "5ec6b22bcc1d916da4e3442d29144c13"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.017769 25359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:38.050338 25359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:38.052495 25359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:38.052644 25359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:38.056712 25359 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.62:42035
I20260808 22:54:38.056744 25462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.62:42035 every 8 connection(s)
I20260808 22:54:38.057036 25359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/data/info.pb
I20260808 22:54:38.057896 25463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:38.059279 25463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:38.063954 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25359
I20260808 22:54:38.064071 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-0/wal/instance
I20260808 22:54:38.064249 22648 external_mini_cluster.cc:1428] Setting key e9949ec0a8edee13c4b2e91b0e15e39e
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:38.068198 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.059940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.22.30.62:42035)
0808 22:54:38.060102 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.060110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.060247 (+ 137us) server_negotiation.cc:407] Connection header received
0808 22:54:38.060761 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.060763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.060792 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.060883 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.061390 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.062113 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.062972 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.063241 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.063341 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.065319 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.065325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.065325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.065331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.066773 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.067143 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.067146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.067146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.067213 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.067450 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.067450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.067451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.067618 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.067698 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.067859 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.067976 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260808 22:54:38.068202 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.059906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42035 (local address 127.0.0.1:58078)
0808 22:54:38.060093 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.060149 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.060611 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.060889 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.060891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.061128 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.061284 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.061288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.062215 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.062216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.062865 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.062868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.063343 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.063344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.063442 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.063847 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.063854 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.065213 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.066866 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.066872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.066874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.067050 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.067286 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.067286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.067286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.067361 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.067683 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.067687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.067740 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.067836 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.068032 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":82,"thread_start_us":84,"threads_started":1}
I20260808 22:54:38.068821 25463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:38.069221 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.069050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44370)
0808 22:54:38.069113 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.069190 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 22:54:38.069375 25384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
I20260808 22:54:38.069795 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.069678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44374)
0808 22:54:38.069734 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.069766 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260808 22:54:38.069913 25463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:37863: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.22.30.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab.
Entry for principal kudu/127.22.30.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab.
I20260808 22:54:38.087363 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:37863
--webserver_interface=127.22.30.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab
--principal=kudu/127.22.30.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:42035,127.22.30.61:37863,127.22.30.60:37997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:54:38.119130 25463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } attempt: 1
I20260808 22:54:38.119516 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.119410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44384)
0808 22:54:38.119446 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.119477 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260808 22:54:38.119895 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.119747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44400)
0808 22:54:38.119814 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.119853 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 22:54:38.120028 25463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:37863: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
W20260808 22:54:38.160962 25473 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:54:38.161132 25473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:38.161149 25473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:38.161209 25473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:38.162256 25473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:38.162297 25473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:38.162310 25473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:38.162350 25473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:38.162374 25473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:38.163769 25473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:42035,127.22.30.61:37863,127.22.30.60:37997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.61.keytab
--principal=kudu/127.22.30.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.61:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.30.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:38.163995 25473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:38.164201 25473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:54:38.168784 25473 server_base.cc:1034] running on GCE node
W20260808 22:54:38.168790 25480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:38.168762 25483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:38.168754 25481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:38.169116 25473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:38.169382 25473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:38.170516 25473 hybrid_clock.cc:648] HybridClock initialized: now 1786229678170503 us; error 29 us; skew 500 ppm
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:38.172793 25473 init.cc:377] Logged in from keytab as kudu/127.22.30.61@KRBTEST.COM (short username kudu)
I20260808 22:54:38.173231 25473 webserver.cc:460] Webserver started at http://127.22.30.61:35757/ using document root <none> and password file <none>
I20260808 22:54:38.173488 25473 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:38.173537 25473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:38.173699 25473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:38.174647 25473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data/instance:
uuid: "0eee2823f9c94aaf8b261fb077e2c833"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "5e40d1a8051f3eebe9f6b0b151b6dc91"
server_key_iv: "ad53fb28ba05a27dd864f4205ac96636"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.174971 25473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal/instance:
uuid: "0eee2823f9c94aaf8b261fb077e2c833"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "5e40d1a8051f3eebe9f6b0b151b6dc91"
server_key_iv: "ad53fb28ba05a27dd864f4205ac96636"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.176183 25473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 22:54:38.176805 25493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:38.176964 25473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 22:54:38.177042 25473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal
uuid: "0eee2823f9c94aaf8b261fb077e2c833"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "5e40d1a8051f3eebe9f6b0b151b6dc91"
server_key_iv: "ad53fb28ba05a27dd864f4205ac96636"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.177109 25473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:38.188158 25463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } attempt: 2
I20260808 22:54:38.188480 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.188366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44402)
0808 22:54:38.188423 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.188453 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 22:54:38.188830 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.188704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44406)
0808 22:54:38.188744 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.188794 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 22:54:38.188956 25463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.61:37863: Network error: Client connection negotiation failed: client connection to 127.22.30.61:37863: connect: Connection refused (error 111)
I20260808 22:54:38.200604 25473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:38.202016 25473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:38.202165 25473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:38.205901 25473 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.61:37863
I20260808 22:54:38.205928 25573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.61:37863 every 8 connection(s)
I20260808 22:54:38.206243 25473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/data/info.pb
I20260808 22:54:38.207105 25575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:38.208532 25575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:38.211467 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25473
I20260808 22:54:38.211589 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-1/wal/instance
I20260808 22:54:38.211805 22648 external_mini_cluster.cc:1428] Setting key 746afb822f3514c1c3dc9a9b7b9cf6bb
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:38.216960 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.209084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.22.30.62:42035)
0808 22:54:38.209134 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.209136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.209318 (+ 182us) server_negotiation.cc:407] Connection header received
0808 22:54:38.209724 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.209725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.209734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.209785 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.210289 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.210777 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.211905 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.212233 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.212308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.214365 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.214366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.214367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.214373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.215688 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.216074 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.216075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.216076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.216130 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.216376 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.216377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.216377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.216501 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.216565 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.216814 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.216889 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 22:54:38.217121 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.209040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42035 (local address 127.0.0.1:58082)
0808 22:54:38.209194 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.209231 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.209650 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.209838 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.209840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.210048 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.210220 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.210223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.210877 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.210878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.211797 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.211799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.212330 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.212331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.212430 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.212877 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.212884 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.214259 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.215781 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.215789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.215790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.216001 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.216191 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.216192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.216193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.216298 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.216571 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.216579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.216691 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.216776 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.216903 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260808 22:54:38.217708 25575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.30.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.30.60@KRBTEST.COM" created.
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:54:38.226958 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.217912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44422)
0808 22:54:38.217952 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.217962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.217997 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.218266 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.218266 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.218367 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.218408 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.218409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.219649 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.219650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.220151 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.220153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.220919 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.220920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.221008 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.221370 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.221374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.224177 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.225877 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.225878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.225879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.226081 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.226309 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.226310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.226310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.226363 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.226660 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.226660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.226694 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.226776 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.226864 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260808 22:54:38.227021 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.217959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.22.30.61:37863)
0808 22:54:38.218114 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.218118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.218142 (+ 24us) server_negotiation.cc:407] Connection header received
0808 22:54:38.218164 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.218166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.218180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.218257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.218477 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.219577 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.220226 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.220710 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.220912 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.224264 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.224272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.224272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.224279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.225797 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.226169 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.226172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.226172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.226230 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.226446 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.226447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.226447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.226569 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.226673 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.226825 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.226912 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260808 22:54:38.227691 25575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:38.228011 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.227914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41852)
0808 22:54:38.227960 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.227996 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 22:54:38.228169 25497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
I20260808 22:54:38.228530 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.228452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41854)
0808 22:54:38.228497 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.228517 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 22:54:38.228646 25575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:37997: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.30.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab.
Entry for principal kudu/127.22.30.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab.
I20260808 22:54:38.237685 22648 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskT1Dm48/build/release/bin/kudu
/tmp/dist-test-taskT1Dm48/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:37997
--webserver_interface=127.22.30.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab
--principal=kudu/127.22.30.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.30.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.30.62:42035,127.22.30.61:37863,127.22.30.60:37997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 22:54:38.277976 25575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } attempt: 1
I20260808 22:54:38.278379 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.278259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41870)
0808 22:54:38.278313 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.278360 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 22:54:38.278728 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.278645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41872)
0808 22:54:38.278687 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.278709 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 22:54:38.278860 25575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:37997: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
I20260808 22:54:38.280105 25463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } attempt: 3
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
I20260808 22:54:38.286586 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.280453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.22.30.61:37863)
0808 22:54:38.280488 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.280490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.280499 (+ 9us) server_negotiation.cc:407] Connection header received
0808 22:54:38.280554 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.280555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.280566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.280613 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.280909 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.281368 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.282019 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.282229 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.282330 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.284264 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.284265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.284265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.284271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.285612 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.285967 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.285968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.285968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.286013 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.286201 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.286201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.286201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.286290 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.286334 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.286486 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.286541 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:38.286635 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.280393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44424)
0808 22:54:38.280445 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.280455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.280498 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.280627 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.280628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.280763 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.280824 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.280826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.281452 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.281453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.281942 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.281943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.282331 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.282332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.282420 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.282824 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.282828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.284183 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.285692 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.285693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.285694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.285896 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.286073 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.286074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.286075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.286145 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.286350 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.286351 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.286396 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.286464 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.286551 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:54:38.287043 25463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:38.287412 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.287276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41888)
0808 22:54:38.287331 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.287365 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 22:54:38.287684 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.287597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41900)
0808 22:54:38.287645 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.287669 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260808 22:54:38.287770 25463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:37997: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
W20260808 22:54:38.315321 25582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 22:54:38.315552 25582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 22:54:38.315568 25582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 22:54:38.315586 25582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 22:54:38.316644 25582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 22:54:38.316679 25582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 22:54:38.316704 25582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 22:54:38.316716 25582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 22:54:38.316727 25582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 22:54:38.318053 25582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.30.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.30.62:42035,127.22.30.61:37863,127.22.30.60:37997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/krb5kdc/kudu_127.22.30.60.keytab
--principal=kudu/127.22.30.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.30.60:37997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.30.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 21:59:43 UTC on ccycloud.quasar-plhyte.root.comops.site
build id 4575490
I20260808 22:54:38.318209 25582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 22:54:38.318380 25582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 22:54:38.318895 25463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } attempt: 1
I20260808 22:54:38.319289 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.319174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41902)
0808 22:54:38.319230 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.319260 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 22:54:38.319661 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.319574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41918)
0808 22:54:38.319607 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.319637 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260808 22:54:38.319792 25463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:37997: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
W20260808 22:54:38.323048 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 22:54:38.323020 25589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:38.323137 25582 server_base.cc:1034] running on GCE node
W20260808 22:54:38.323014 25596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 22:54:38.323484 25582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 22:54:38.323745 25582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 22:54:38.324905 25582 hybrid_clock.cc:648] HybridClock initialized: now 1786229678324890 us; error 41 us; skew 500 ppm
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 22:54:38.327462 25582 init.cc:377] Logged in from keytab as kudu/127.22.30.60@KRBTEST.COM (short username kudu)
I20260808 22:54:38.327884 25582 webserver.cc:460] Webserver started at http://127.22.30.60:40179/ using document root <none> and password file <none>
I20260808 22:54:38.328127 25582 fs_manager.cc:359] Metadata directory not provided
I20260808 22:54:38.328229 25582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 22:54:38.328392 25582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 22:54:38.329348 25582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data/instance:
uuid: "413c8a0f5cb842f69eff84c7778a0fae"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "ea2f5b56e0dbe9f189f92e66c6ace614"
server_key_iv: "df81e9f88c1d74b9d65c350c24f4b39e"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.329694 25582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal/instance:
uuid: "413c8a0f5cb842f69eff84c7778a0fae"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "ea2f5b56e0dbe9f189f92e66c6ace614"
server_key_iv: "df81e9f88c1d74b9d65c350c24f4b39e"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.330916 25582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 22:54:38.331708 25603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 22:54:38.331894 25582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 22:54:38.331993 25582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data,/tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal
uuid: "413c8a0f5cb842f69eff84c7778a0fae"
format_stamp: "Formatted at 2026-08-08 22:54:38 on dist-test-slave-x4qh"
server_key: "ea2f5b56e0dbe9f189f92e66c6ace614"
server_key_iv: "df81e9f88c1d74b9d65c350c24f4b39e"
server_key_version: "encryptionkey@0"
I20260808 22:54:38.332074 25582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 22:54:38.346438 25582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 22:54:38.347002 25575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } attempt: 2
I20260808 22:54:38.347412 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.347259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41930)
0808 22:54:38.347331 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.347365 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 22:54:38.347801 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.347699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41946)
0808 22:54:38.347748 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.347776 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 22:54:38.347944 25575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.30.60:37997: Network error: Client connection negotiation failed: client connection to 127.22.30.60:37997: connect: Connection refused (error 111)
I20260808 22:54:38.348596 25582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 22:54:38.348766 25582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 22:54:38.352615 25582 rpc_server.cc:307] RPC server started. Bound to: 127.22.30.60:37997
I20260808 22:54:38.352644 25686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.30.60:37997 every 8 connection(s)
I20260808 22:54:38.352980 25582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/data/info.pb
I20260808 22:54:38.353981 25688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 22:54:38.355365 25688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:38.362131 22648 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskT1Dm48/build/release/bin/kudu as pid 25582
I20260808 22:54:38.362246 22648 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskT1Dm48/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786229410580610-22648-0/minicluster-data/master-2/wal/instance
I20260808 22:54:38.362474 22648 external_mini_cluster.cc:1428] Setting key c005717ccaf1c3dba3d3044cec86cc3e
I20260808 22:54:38.363471 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.356016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.22.30.62:42035)
0808 22:54:38.356072 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.356074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.356265 (+ 191us) server_negotiation.cc:407] Connection header received
0808 22:54:38.356714 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.356716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.356727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.356795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.357331 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.357770 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.358665 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.358870 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.358948 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.360831 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.360833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.360833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.360838 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.361895 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.362351 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.362353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.362353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.362414 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.362707 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.362708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.362708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.362849 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.362920 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.363256 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.363351 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:54:38.363571 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.355980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.62:42035 (local address 127.0.0.1:58084)
0808 22:54:38.356127 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.356176 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.356621 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.356855 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.356859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.357077 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.357235 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.357239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.357859 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.357860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.358558 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.358560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.358967 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.358968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.359087 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.359544 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.359552 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.360746 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.361977 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.361984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.361986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.362267 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.362475 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.362476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.362476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.362603 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.362920 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.362925 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.363090 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.363208 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.363344 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260808 22:54:38.364125 25688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:54:38.371203 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.364377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.61:37863 (local address 127.0.0.1:44428)
0808 22:54:38.364483 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.364493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.364543 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.364651 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.364651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.364780 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.364826 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.364827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.365655 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.365656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.366311 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.366313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.366806 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.366807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.366902 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.367282 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.367287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.368482 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.370176 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.370177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.370177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.370439 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.370628 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.370629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.370629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.370691 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.370914 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.370915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.370960 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.371042 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.371128 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 22:54:38.371239 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.364438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.22.30.61:37863)
0808 22:54:38.364497 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.364498 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.364530 (+ 32us) server_negotiation.cc:407] Connection header received
0808 22:54:38.364598 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.364599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.364606 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.364647 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.364883 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.365580 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.366423 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.366722 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.366775 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.368562 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.368563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.368564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.368569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.370085 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.370517 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.370517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.370518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.370565 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.370756 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.370757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.370757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.370849 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.370898 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.371057 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.371153 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 22:54:38.371615 25688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } has no permanent_uuid. Determining permanent_uuid...
I20260808 22:54:38.372409 25692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.362864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.22.30.60:37997)
0808 22:54:38.363013 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.363017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.363106 (+ 89us) server_negotiation.cc:407] Connection header received
0808 22:54:38.363127 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.363129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.363142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.363216 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.363986 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.364932 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.365927 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.366356 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.366428 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.369198 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.369205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.369205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.369212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.371036 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.371459 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.371462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.371462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.371516 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.371772 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.371773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.371773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.371948 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.372091 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.372170 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.372304 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.60@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.62@KRBTEST.COM
I20260808 22:54:38.378683 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.371843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41964)
0808 22:54:38.371892 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.371902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.371943 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.372136 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.372137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.372290 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.372338 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.372340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.373161 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.373161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.373665 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.373667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.374041 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.374042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.374088 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.374458 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.374464 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.375552 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.377601 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.377602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.377602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.377849 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.378061 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.378062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.378063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.378125 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.378390 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.378391 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.378439 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.378514 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.378611 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 22:54:38.378935 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.371876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.22.30.60:37997)
0808 22:54:38.372022 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.372024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.372030 (+ 6us) server_negotiation.cc:407] Connection header received
0808 22:54:38.372046 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.372046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.372054 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.372135 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.372425 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.373087 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.373763 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.373960 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.374043 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.375631 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.375633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.375633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.375639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.377520 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.377940 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.377941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.377941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.377993 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.378187 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.378189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.378189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.378315 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.378376 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.378543 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.378670 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260808 22:54:38.380228 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.373092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.22.30.62:42035)
0808 22:54:38.373131 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.373133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.373141 (+ 8us) server_negotiation.cc:407] Connection header received
0808 22:54:38.373194 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.373195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.373203 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.373256 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.373686 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.374102 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.374708 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.374965 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.375029 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.377341 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.377342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.377342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.377348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.379019 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.379498 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.379499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.379499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.379566 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.379723 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.379724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.379724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.379838 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.380008 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.380045 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.380121 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 22:54:38.380375 25688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: Bootstrap starting.
I20260808 22:54:38.380927 25688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:38.381187 25688 log.cc:824] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:38.381944 25688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: No bootstrap required, opened a new log
I20260808 22:54:38.383173 25688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.383299 25688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413c8a0f5cb842f69eff84c7778a0fae, State: Initialized, Role: FOLLOWER
I20260808 22:54:38.383376 25688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.383723 25688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:38.383692 25695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.383783 25695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: This master's current role is: FOLLOWER
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.30.61@KRBTEST.COM
W20260808 22:54:38.385758 25709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:38.385833 25709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:38.385854 25582 master_runner.cc:186] Error getting master registration for 127.22.30.62:42035: OK, instance_id { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" instance_seqno: 1786229678055630 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:54:38.388144 25582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:54:38.388818 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.381843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.22.30.61:37863)
0808 22:54:38.381884 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.381886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.381905 (+ 19us) server_negotiation.cc:407] Connection header received
0808 22:54:38.381923 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.381924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.381931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.381976 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.382433 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.382841 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.383556 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.383848 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.383909 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.386215 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.386216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.386216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.386223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.387763 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.388120 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.388121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.388121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.388174 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.388340 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.388341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.388341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.388442 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.388593 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.388626 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.388690 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 22:54:38.390112 22648 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 22:54:38.391158 25421 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58098
I20260808 22:54:38.394909 25463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } attempt: 2
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.62@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:54:38.401592 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.395147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.22.30.60:37997)
0808 22:54:38.395178 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.395180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.395187 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:38.395252 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.395253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.395260 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.395309 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.395577 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.396171 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.396978 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.397283 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.397366 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.399152 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.399153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.399154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.399160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.400547 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.400866 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.400867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.400868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.400920 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.401102 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.401103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.401104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.401225 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.401307 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.401456 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.401519 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 22:54:38.401597 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.395106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:41970)
0808 22:54:38.395143 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.395151 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.395179 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.395327 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.395328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.395476 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.395510 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.395512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.396242 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.396243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.396911 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.396912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.397374 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.397375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.397429 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.397816 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.397820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.399069 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.400618 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.400619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.400620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.400765 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.400990 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.400991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.400992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.401040 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.401304 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.401305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.401342 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.401433 (+ 91us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.401515 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260808 22:54:38.402782 25463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc: Bootstrap starting.
I20260808 22:54:38.403424 25463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:38.403710 25463 log.cc:824] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:38.404420 25463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc: No bootstrap required, opened a new log
I20260808 22:54:38.405367 25463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.405464 25463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3c6ae6e9c547e59f7945ca850b5cbc, State: Initialized, Role: FOLLOWER
I20260808 22:54:38.405557 25463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.405804 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.405864 25463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:38.405879 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:38.405987 25421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 22:54:38.406059 25421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:38.406795 25421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.407471 25421 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [CANDIDATE]: Term 1 election: Requested vote from peers 0eee2823f9c94aaf8b261fb077e2c833 (127.22.30.61:37863), 413c8a0f5cb842f69eff84c7778a0fae (127.22.30.60:37997)
W20260808 22:54:38.407708 25734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 22:54:38.407781 25734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:38.407908 25642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "413c8a0f5cb842f69eff84c7778a0fae"
I20260808 22:54:38.408002 25642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:38.408919 25642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3c6ae6e9c547e59f7945ca850b5cbc in term 1.
I20260808 22:54:38.409142 25385 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413c8a0f5cb842f69eff84c7778a0fae, 9f3c6ae6e9c547e59f7945ca850b5cbc; no voters:
I20260808 22:54:38.409253 25712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 22:54:38.409399 25712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [term 1 LEADER]: Becoming Leader. State: Replica: 9f3c6ae6e9c547e59f7945ca850b5cbc, State: Running, Role: LEADER
I20260808 22:54:38.409487 25712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.409879 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3c6ae6e9c547e59f7945ca850b5cbc. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.409976 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:38.409857 25359 master_runner.cc:186] Error getting master registration for 127.22.30.61:37863: OK, instance_id { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" instance_seqno: 1786229678204833 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 22:54:38.411576 25359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 22:54:38.419881 25739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 22:54:38.420102 25739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 22:54:38.420732 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.412602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.22.30.60:37997)
0808 22:54:38.412653 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.412655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.412798 (+ 143us) server_negotiation.cc:407] Connection header received
0808 22:54:38.412898 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.412899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.412908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.412960 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.413689 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.414417 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.415406 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.415691 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.415745 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.417909 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.417910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.417910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.417915 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.419535 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.419907 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.419908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.419908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.419968 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.420197 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.420198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.420199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.420312 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.420406 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.420547 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.420635 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 22:54:38.425539 25642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [term 1 FOLLOWER]: Refusing update from remote peer 9f3c6ae6e9c547e59f7945ca850b5cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:38.425798 25736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 22:54:38.427143 25695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3c6ae6e9c547e59f7945ca850b5cbc. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.427224 25695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:38.432348 25736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.432430 25736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:38.432819 25695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.432917 25695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:38.432353 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.424070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.22.30.62:42035)
0808 22:54:38.424105 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.424107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.424258 (+ 151us) server_negotiation.cc:407] Connection header received
0808 22:54:38.424542 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.424543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.424555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.424585 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.425201 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.425808 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.427130 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.427402 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.427440 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.428827 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.428828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.428828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.428835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.430544 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.431083 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.431097 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.431098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.431149 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.431486 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.431487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.431487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.431603 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.431690 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.432172 (+ 482us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.432241 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:54:38.433104 25748 mvcc.cc:205] Tried to move back new op lower bound from 7316396762823602176 to 7316396762765901824. Current Snapshot: MvccSnapshot[applied={T|T < 7316396762823602176}]
I20260808 22:54:38.433487 25750 mvcc.cc:205] Tried to move back new op lower bound from 7316396762823602176 to 7316396762765901824. Current Snapshot: MvccSnapshot[applied={T|T < 7316396762823602176}]
I20260808 22:54:38.434015 25739 catalog_manager.cc:1347] Generated new cluster ID: c90d6267fb474f6cac15bac5ca33136e
I20260808 22:54:38.434063 25739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 22:54:38.437609 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.424585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.22.30.61:37863)
0808 22:54:38.424626 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.424628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.427770 (+ 3142us) server_negotiation.cc:407] Connection header received
0808 22:54:38.427894 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.427895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.427906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.427936 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.429428 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.430133 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.431725 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.431995 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.432033 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.434790 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.434791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.434791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.434797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.436394 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.436811 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.436812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.436812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.436860 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.437104 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.437106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.437106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.437214 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.437284 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.437450 (+ 166us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.437522 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 22:54:38.439129 25575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } attempt: 3
Aug 08 22:54:38 dist-test-slave-x4qh krb5kdc[25340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786229678, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.30.61@KRBTEST.COM for kudu/127.22.30.60@KRBTEST.COM
I20260808 22:54:38.446028 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.439440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.22.30.60:37997)
0808 22:54:38.439488 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.439490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.439497 (+ 7us) server_negotiation.cc:407] Connection header received
0808 22:54:38.439516 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.439516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.439524 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.439584 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.439847 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.440545 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.441183 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.441493 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.441551 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.443317 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 22:54:38.443319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 22:54:38.443319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 22:54:38.443325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 22:54:38.444969 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.445332 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.445333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.445334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.445394 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 22:54:38.445579 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 22:54:38.445580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 22:54:38.445581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 22:54:38.445693 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 22:54:38.445743 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:38.445892 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 22:54:38.445950 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 22:54:38.446028 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:38.439338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.30.60:37997 (local address 127.0.0.1:42000)
0808 22:54:38.439394 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 22:54:38.439404 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 22:54:38.439442 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 22:54:38.439592 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 22:54:38.439593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 22:54:38.439727 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0808 22:54:38.439756 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.439757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.440606 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.440607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.441102 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 22:54:38.441104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 22:54:38.441552 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 22:54:38.441552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 22:54:38.441615 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:38.441979 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 22:54:38.441983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 22:54:38.443225 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 22:54:38.445053 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.445054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.445054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.445250 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.445447 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 22:54:38.445465 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 22:54:38.445465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 22:54:38.445519 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 22:54:38.445738 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 22:54:38.445738 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 22:54:38.445797 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 22:54:38.445858 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 22:54:38.445940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 22:54:38.447095 25575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: Bootstrap starting.
I20260808 22:54:38.447844 25575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: Neither blocks nor log segments found. Creating new log.
I20260808 22:54:38.448184 25575 log.cc:824] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: Log is configured to *not* fsync() on all Append() calls
I20260808 22:54:38.449026 25575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: No bootstrap required, opened a new log
I20260808 22:54:38.450868 25575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.450981 25575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eee2823f9c94aaf8b261fb077e2c833, State: Initialized, Role: FOLLOWER
I20260808 22:54:38.451059 25575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } }
I20260808 22:54:38.451390 25531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0eee2823f9c94aaf8b261fb077e2c833"
I20260808 22:54:38.451435 25528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [term 0 FOLLOWER]: Advancing to term 1
I20260808 22:54:38.451313 25755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.451860 25755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:38.452201 25528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [term 1 FOLLOWER]: Refusing update from remote peer 9f3c6ae6e9c547e59f7945ca850b5cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 22:54:38.452528 25712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 22:54:38.452680 25500 tablet.cc:2369] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 22:54:38.451350 25575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 22:54:38.453060 25531 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f3c6ae6e9c547e59f7945ca850b5cbc in term 1.
I20260808 22:54:38.454062 25755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3c6ae6e9c547e59f7945ca850b5cbc. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.454149 25755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:38.454185 25755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.454222 25755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 22:54:38.465142 25774 catalog_manager.cc:1259] Loaded cluster ID: c90d6267fb474f6cac15bac5ca33136e
I20260808 22:54:38.465214 25774 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: loading cluster ID for follower catalog manager: success
W20260808 22:54:38.465308 25774 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260808 22:54:38.465344 25774 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260808 22:54:38.471781 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f3c6ae6e9c547e59f7945ca850b5cbc" member_type: VOTER last_known_addr { host: "127.22.30.62" port: 42035 } } peers { permanent_uuid: "0eee2823f9c94aaf8b261fb077e2c833" member_type: VOTER last_known_addr { host: "127.22.30.61" port: 37863 } } peers { permanent_uuid: "413c8a0f5cb842f69eff84c7778a0fae" member_type: VOTER last_known_addr { host: "127.22.30.60" port: 37997 } } }
I20260808 22:54:38.471866 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc [sys.catalog]: This master's current role is: LEADER
I20260808 22:54:38.473817 25739 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 22:54:38.474445 25739 catalog_manager.cc:1504] Loading token signing keys...
I20260808 22:54:38.481586 25739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f3c6ae6e9c547e59f7945ca850b5cbc: Generated new TSK 0
I20260808 22:54:38.481745 25739 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 22:54:38.506574 25693 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.504827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.22.30.60:37997)
0808 22:54:38.504862 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.504864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.505978 (+ 1114us) server_negotiation.cc:407] Connection header received
0808 22:54:38.506096 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.506097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.506106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.506132 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.506519 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42008: BlockingRecv error: recv got EOF from 127.0.0.1:42008 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 22:54:38.507352 25467 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.505160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.22.30.62:42035)
0808 22:54:38.505193 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.505195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.506832 (+ 1637us) server_negotiation.cc:407] Connection header received
0808 22:54:38.506911 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.506912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.506921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.506945 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.507304 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58106: BlockingRecv error: recv got EOF from 127.0.0.1:58106 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 22:54:38.508015 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.505509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.22.30.61:37863)
0808 22:54:38.505709 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.505710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.507556 (+ 1846us) server_negotiation.cc:407] Connection header received
0808 22:54:38.507666 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.507667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.507675 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.507701 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.507968 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44448: BlockingRecv error: recv got EOF from 127.0.0.1:44448 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
W20260808 22:54:38.510073 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.509406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.22.30.61:37863)
0808 22:54:38.509438 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.509440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.509612 (+ 172us) server_negotiation.cc:407] Connection header received
0808 22:54:38.509708 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.509708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.509723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.509749 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.510039 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44452: BlockingRecv error: recv got EOF from 127.0.0.1:44452 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 22:54:38.512879 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.512238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.22.30.61:37863)
0808 22:54:38.512263 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.512265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.512463 (+ 198us) server_negotiation.cc:407] Connection header received
0808 22:54:38.512540 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.512541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.512548 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.512572 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.512845 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44466: BlockingRecv error: recv got EOF from 127.0.0.1:44466 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260808 22:54:38.516571 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.515919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.22.30.61:37863)
0808 22:54:38.515950 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.515952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.516161 (+ 209us) server_negotiation.cc:407] Connection header received
0808 22:54:38.516240 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.516240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.516248 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.516272 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.516537 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44468: BlockingRecv error: recv got EOF from 127.0.0.1:44468 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 22:54:38.522312 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.521586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.22.30.61:37863)
0808 22:54:38.521615 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.521617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.521857 (+ 240us) server_negotiation.cc:407] Connection header received
0808 22:54:38.521947 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.521948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.521957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.521984 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.522265 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44480: BlockingRecv error: recv got EOF from 127.0.0.1:44480 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260808 22:54:38.532142 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.531361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.22.30.61:37863)
0808 22:54:38.531430 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.531432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.531655 (+ 223us) server_negotiation.cc:407] Connection header received
0808 22:54:38.531749 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.531750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.531759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.531788 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.532094 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44492: BlockingRecv error: recv got EOF from 127.0.0.1:44492 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260808 22:54:38.550181 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.549295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.22.30.61:37863)
0808 22:54:38.549331 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.549333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.549588 (+ 255us) server_negotiation.cc:407] Connection header received
0808 22:54:38.549740 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.549741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.549751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.549780 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.550138 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44504: BlockingRecv error: recv got EOF from 127.0.0.1:44504 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 22:54:38.584239 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.583427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.22.30.61:37863)
0808 22:54:38.583464 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.583466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.583614 (+ 148us) server_negotiation.cc:407] Connection header received
0808 22:54:38.583732 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.583733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.583745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.583819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.584184 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44520: BlockingRecv error: recv got EOF from 127.0.0.1:44520 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 22:54:38.649922 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.649294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.22.30.61:37863)
0808 22:54:38.649380 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.649382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.649395 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:54:38.649466 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.649466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.649476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.649526 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.649865 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44524: BlockingRecv error: recv got EOF from 127.0.0.1:44524 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260808 22:54:38.779719 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:38.779043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.22.30.61:37863)
0808 22:54:38.779117 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:38.779119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:38.779216 (+ 97us) server_negotiation.cc:407] Connection header received
0808 22:54:38.779284 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:38.779285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:38.779295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:38.779338 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:38.779676 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44538: BlockingRecv error: recv got EOF from 127.0.0.1:44538 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260808 22:54:39.037361 25579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 22:54:39.036816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.22.30.61:37863)
0808 22:54:39.036889 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:39.036891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:39.036903 (+ 12us) server_negotiation.cc:407] Connection header received
0808 22:54:39.036955 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:39.036956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:39.036966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:39.037015 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 22:54:39.037302 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44544: BlockingRecv error: recv got EOF from 127.0.0.1:44544 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 22:54:39.386236 25709 catalog_manager.cc:1259] Loaded cluster ID: c90d6267fb474f6cac15bac5ca33136e
I20260808 22:54:39.386317 25709 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: loading cluster ID for follower catalog manager: success
I20260808 22:54:39.387082 25709 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: acquiring CA information for follower catalog manager: success
I20260808 22:54:39.387414 25709 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 413c8a0f5cb842f69eff84c7778a0fae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:54:39.466374 25774 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: acquiring CA information for follower catalog manager: success
I20260808 22:54:39.466750 25774 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0eee2823f9c94aaf8b261fb077e2c833: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 22:54:39.552989 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:39.550529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.22.30.61:37863)
0808 22:54:39.550646 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:39.550651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:39.550664 (+ 13us) server_negotiation.cc:407] Connection header received
0808 22:54:39.550733 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:39.550734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:39.550783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:39.550856 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:39.551002 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.551524 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.552237 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.552421 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.552491 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:39.552652 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:39.552757 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:39.552840 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:39.552857 (+ 17us) server_negotiation.cc:299] Negotiation successful
0808 22:54:39.552877 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260808 22:54:39.556983 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:39.554731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.22.30.60:37997)
0808 22:54:39.554879 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:39.554881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:39.554904 (+ 23us) server_negotiation.cc:407] Connection header received
0808 22:54:39.554938 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:39.554938 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:39.554971 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:39.555051 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:39.555196 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.555704 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.556350 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.556519 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.556569 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:39.556697 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:39.556786 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:39.556831 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:39.556867 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 22:54:39.556882 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260808 22:54:39.561080 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:39.558579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.22.30.60:37997)
0808 22:54:39.558655 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:39.558657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:39.558798 (+ 141us) server_negotiation.cc:407] Connection header received
0808 22:54:39.558823 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:39.558824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:39.558836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:39.558890 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:39.558998 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.559542 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.560441 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.560702 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.560765 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:39.560862 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:39.560886 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:39.560927 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:39.560991 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 22:54:39.561010 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 22:54:39.561553 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:39.558651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.22.30.61:37863)
0808 22:54:39.558691 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:39.558693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:39.558826 (+ 133us) server_negotiation.cc:407] Connection header received
0808 22:54:39.558944 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:39.558944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:39.558952 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:39.558978 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:39.559168 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.559868 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.560717 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.560987 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.561039 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:39.561299 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:39.561329 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:39.561355 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:39.561466 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 22:54:39.561484 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 22:54:39.561868 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 22:54:39.558478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.22.30.62:42035)
0808 22:54:39.558578 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 22:54:39.558580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 22:54:39.558728 (+ 148us) server_negotiation.cc:407] Connection header received
0808 22:54:39.558771 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 22:54:39.558772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 22:54:39.558829 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 22:54:39.558892 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 22:54:39.559051 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.559810 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.560686 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 22:54:39.560999 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 22:54:39.561057 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 22:54:39.561288 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 22:54:39.561390 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 22:54:39.561449 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 22:54:39.561737 (+ 288us) server_negotiation.cc:299] Negotiation successful
0808 22:54:39.561784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260808 22:54:39.563304 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25359
I20260808 22:54:39.576864 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25473
I20260808 22:54:39.584348 22648 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskT1Dm48/build/release/bin/kudu with pid 25582
2026-08-08T22:54:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2928 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5847 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269011 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST